LLM systems
for revenue
and analytics

case · sales and traffic

business request"Find the reasons behind the May conversion drop and the shift in plan mix"

Conversion fell somewhere else than expected

Every substantive conversation for a product line across three months: unanswered and empty calls were excluded from the base, not diluted into percentages. Dozens of features per deal — need, purchase readiness, plan, objections and how they were handled, competitors mentioned, rep behaviour, coverage of new pitch blocks.

The month-over-month view showed the problem was not the sales team: inbound traffic was cooling, the share of purchase-ready customers was falling, money objections were growing. Observations are separated from explanations — 'the share of hot leads fell' is a fact, 'therefore revenue dropped' is a hypothesis, labelled as one, with a note on how to test it.

→ decisions on traffic buying, script and pricing pitch.

case · product and pricing

business request"Reasons for refusal, and whether customers ask for further-education programmes"

"Too expensive" is not always about price

Records that did not pass through the service were excluded explicitly — with the loss stated in the report rather than quietly dropped. The sample is defined by a filter: conversations over five minutes on a specific product line. Reasons for no purchase are broken down on two levels: a top-level category first, then a second-level breakdown of the two largest groups.

The second level changes the decision. 'Cost' at the top looks like a discount request, but inside the group something else dominates — the customer cannot get instalment financing. That is not a pricing problem, it is a payment-instrument problem. Every category is backed by a customer quote, so the conclusion can be re-checked instead of taken on trust.

→ payment scenarios, handling financing rejections, and a separate demand assessment for programmes with a formal diploma.

case · operations and lost revenue

business request"Understand why existing customers contact sales and find growth points in the customer journey"

Part of the sales team's load is not sales

Every deal closed under the 'existing customer' reason across several months. The report states explicitly how many requests reached the calculation and what did not. Each request is classified by meaning and by owner: sales, refunds or support, plus a month-over-month view to tell a one-off spike from a persistent load.

Separately I looked for lost sales: cases where the customer actually wanted to buy something new, but the deal was closed as 'existing customer'. It also turned out that a significant part of the inbound flow is service questions and work with money already paid. That is not only load on the wrong team: while those requests sit in the denominator, the team's conversion rate is measured wrong.

→ routing requests between teams, returning lost deals to sales, and a correct conversion denominator.

stt

A managed speech service instead of self-hosted Whisper

I will revisit it when volume makes the price gap larger than the cost of operations: then a hybrid — bulk traffic in-house, hard recordings in the cloud.

→ the runtime where this latency is budgeted stage by stage

retrieval

pgvector instead of a dedicated vector database

The migration threshold is set in advance: when retrieval time becomes noticeable next to model time, or the index stops fitting in memory.

→ retrieval strategies compared at different corpus sizes

evaluation

Model scoring against a frozen label set

The price is judge drift when the model version changes. That is why the set is frozen and every comparison runs on the same labels.

→ how a set is frozen and how tampering is caught

migration to a cheaper model

What breaks first and how it is caught

When a scenario moves to a model several times cheaper, overall quality is not what falls first. Long context and format stability break first: invalid output schemas appear more often, and recall drops on rare critical categories. Average accuracy looks fine and shows nothing.

So the release is gated on two things: schema validity and recall on critical violations. The fixes are context compression, moving part of the rules from the prompt into code, examples for the failing classes, and a separate pass for critical categories.

The result is the same validated output at −88% scenario cost. Where the cost of a mistake is high, a human stays in the loop: a deliberate limit, not an unfinished feature.

→ the metrics and gates this was checked against

voice ai

45–50%

of eligible routine consultations run without an operator

I turned the contact-centre policy into a live dialogue: script branching, clarifications, barge-in, answers from the knowledge base. Connected to Sipuni telephony, it holds 10 concurrent lines, works around the clock and does not depend on which rep picked up.

Latency is the binding constraint of the channel: 3 seconds to the first reply, up to 10 seconds between replies. It is the sum of speech recognition, one model call carrying the script and knowledge-base context, and speech synthesis. It shrinks not by swapping the model but by how early recognition starts and how much context the request carries.

Barge-in and dialogue-branch control live in the system prompt rather than a separate state machine. Policy edits ship without a release, but the whole dialogue policy sits in one contract — so its changes are checked against the reference set exactly like code.

Hence ×1.2 qualified leads in month one and a saving from 1M RUB per month: more answered calls, round-the-clock coverage, full script coverage and consistent product knowledge.

→ the runtime behind a dialogue like this: stages, barge-in, latency budget

enterprise rag

100+

daily internal users

Two Dify assistants answer staff questions from internal documents — that is what RAG means: the model answers from retrieved fragments rather than from memory. One for sales, one over the corporate Confluence base, 37 documents, purely vector retrieval.

I owned chunking, context assembly, grounding and answer quality. An employee gets an answer with a source link in under 10 seconds instead of roughly five minutes of manual search.

→ retrieval strategies compared and answer grounding checked

Hybrid retrieval is not needed here: keyword matching pays off once queries carry exact part numbers, version strings and rare terms — on a corpus this size it would add complexity without a result. The real work is not the model but document preparation, and making the assistant honestly say 'this is not in the documents'.

llm evaluation

4

task-specific golden sets

Separate sets and metrics for scoring, summaries, semantic analytics and violations. Human calibration, regressions and error analysis after every model change. 98% decision agreement — on a frozen set of 500 deals.

→ methodology and code in the public repository

model economics

−88%

of the key LLM scenario cost

I moved the same validated output from an expensive model to a compact one through prompt, context and pipeline work. The vendor price gap is only half the story: without context compression and a reworked output schema the cheap model does not pass the gates.

I measure cost per 1,000 deals and freeze a dated pricing snapshot — otherwise a before/after comparison means nothing.

evaluating llm output

llm-evaluation-harness

Four task families, a typed error taxonomy, bootstrap intervals and release gates. Critical recall is gated separately from average accuracy — those are exactly the misses an average hides.

→ open the repository · about 2,300 lines

tests and CI on Python 3.11 and 3.12

reference sets

golden-set-builder

Stratified sampling that reports which cells the pool could not fill. Annotator agreement is computed before any model metric, because it is the ceiling. Adjudication without majority vote, and freezing with a hash per record.

→ open the repository · about 1,500 lines

tests and CI on Python 3.11 and 3.12

prompts as contracts

prompt-contracts

Every prompt ships an output schema, refusal rules, a size budget and the golden set that gates its changes. The diff command answers the only question a reviewer has: must evaluation be re-run before merging.

→ open the repository · about 850 lines

tests and CI on Python 3.11 and 3.12

voice runtime

voice-agent-runtime

The dialogue policy is a transition table, not a paragraph in the prompt. A barge-in never resumes the interrupted sentence. Latency is budgeted per stage and judged at p95 — the value only one turn in twenty exceeds — against 3 seconds to the first reply and 10 seconds per turn.

→ open the repository · about 1,100 lines

tests and CI on Python 3.11 and 3.12

retrieval and grounding

rag-grounding-lab

What chunking costs and where each retrieval strategy loses — three failures reproduced rather than hidden. Answers are checked sentence by sentence: an invented number is caught separately from an unsupported phrasing, and an honest refusal counts as grounded.

→ open the repository · about 1,200 lines

tests and CI on Python 3.11 and 3.12

analytics on model output

conversation-intelligence-lab

The methodology from the analytics section, on synthetic data: a strict output contract, contradiction checks, traceable monthly aggregates and a reproducible report.

→ open the repository · about 300 lines

tests and CI on Python 3.11 and 3.12