The world's most accurate retrieval model for qualitative data
AOT is BuildBetter's own retrieval model, built over six years of research and used in production by our customers today. We benchmarked it against RAG, vector search and a model reading all 101 million tokens of a real customer corpus. AOT retrieves 99% of the verified evidence. RAG and vector search retrieve 11%.
Same corpus, same questions, same scoring. Every figure on this page comes from that run, and the method, the caveats and the raw results file are below.
Where search wins, and where it stops working
Vector search and RAG are genuinely excellent technology, and on a large class of problems nothing beats them. They were built to find the best matching passage in a pile of documents, they do that in milliseconds, and they scale to billions of records. If your question has an answer sitting in one place, use them.
Qualitative data breaks that assumption. A customer conversation is not a document with an answer in it; it is one data point in a population. The question is almost never which call mentions this. It is how many, who, and what else came with it. That is a counting problem wearing a search problem's clothes.
The last four rows are the questions product teams actually ask, and they are exactly the rows where returning the k best passages cannot get you there. Not because the search is weak, but because the answer is a property of the whole corpus and the method only ever looks at a slice of it.
The counting, ranking and absence rows are measured on this page. Quote-pulling is a tie on paper and a difference in practice: search hands back passages you still have to read, attribute and sort, while a signal already carries the quote, the speaker, the call it came from, and a type you can filter on, so testimonials come back as testimonials. The change-over-time row rests on architecture rather than a measurement here, because records carry dates and counts while a ranked passage list carries neither. We pulled our own trend tasks from this benchmark when the gold behind them turned out to track corpus growth instead of topic movement.
Two questions that look alike and aren't
"What did Acme say about our pricing on the renewal call?" has an answer that lives in one place. Find the right passage and you are done. This is what search was built for, and search is good at it.
"What are the top five things customers complain about, and how many raised each?" has no single passage that answers it. The evidence is scattered across hundreds of conversations, and the answer is a count over all of them. Miss forty of those conversations and the ranking changes.
The first question needs the best passage. The second needs every passage. Query-time search returns k passages, and k is always smaller than the corpus.
Product teams almost never ask the first kind. They ask what is trending, what is breaking, what customers keep requesting, which segment is loudest about a problem. Every one of those is a question about the whole corpus, and every one of them is where searching at query time runs out of room.
Nobody measures how much their search actually misses
Retrieval usually gets evaluated end to end: ask a question, read the answer, score the answer. That number blends two very different things — how much evidence the system retrieved, and how well a language model wrote it up. Change the prompt and the score moves. We watched exactly that happen in our own runs, where a single sentence in the answer instructions swung whole systems by tiers.
So we split them apart and measured only the bottom half. For every piece of verified customer evidence in a real corpus, we asked each system a question with no judgment in it.
Coverage
Given a verified piece of customer evidence, does the system retrieve it? For query-time search, that means the passage appears in what it returns for that topic. For AOT, it means a signal was extracted from that passage at ingestion. No agent, no answer, no scoring model. Retrieved, or not retrieved.
The corpus, and an answer key that can't be rigged
The corpus is one real workspace, frozen and hashed: 6,018 call recordings and 8,533 support conversations, split into 220,698 passages. About 101 million tokens. No synthetic data.
The answer key had to come from the raw text, never from AOT, or the whole exercise would be circular. Four steps, each one auditable:
Read all 220,698 passages
A model labeled every passage in the corpus against a fixed 14-domain taxonomy written before the run, and grouped what customers raised into themes. Exhaustive, not sampled — a claim about how many customers raised something is worthless if you only looked at a slice.
Verify every piece of evidence with a different lab's model
A model from a different AI lab re-checked all 3,766 evidence-to-theme links: does this passage genuinely support this theme? It agreed 89.4% of the time. Everything it rejected was cut, along with three whole themes that fell below half.
Cut our own voice out of the data
Our support team's own words were being counted as customer themes — the biggest one was our team inviting people to a Slack channel, across 180 conversations. So we classified all 1,349 quotes behind the answer key by whose voice they carried. Only 29.7% were customers. The rest are gone.
Prove the scoring works before scoring anything
A control run gets handed the correct evidence directly and has to score 100%. If the control can't get it right with the answer in hand, no other result means anything. Ours took nine rounds to pass, and four of those rounds found bugs in the answer key rather than the systems.
If a benchmark can't embarrass the team that built it, it isn't measuring anything. This one cost us 23 of our own themes and nine calibration rounds before it produced a single usable number.
The same numbers, with intervals
Figure 1 in exact terms. Intervals are 95% cluster bootstraps that resample themes rather than passages, which is the conservative choice when evidence inside one theme is correlated. Query-time rows are single measurements against a fixed index, so no interval applies.
Full context does not fit: 101 million corpus tokens against a 400,000 token window. Scanning the corpus once for a single question costs about $101 at list price. The two AOT rows bracket the same measurement and the next section explains why the floor is lower than it should be.
More budget doesn't close the gap
The obvious objection to any retrieval result is that the budget was too small. So we swept it: 10, 40, 100, and 400 passages per question, across keyword search, embeddings, and hybrid fusion. Coverage climbs roughly log-linearly. Forty times the reading budget buys about fifteen times the coverage, and the best result is still 27.9%.
Coverage by budget, exact numbers
This is a structural limit of searching at query time, and it has nothing to do with whose embeddings you buy. A question whose answer spans hundreds of conversations cannot be served by a model that returns a fixed number of passages.
How our model retrieves differently
AOT is a retrieval model we built ourselves. It stands for ahead-of-time comprehension, it is proprietary to BuildBetter, and six years of research went into it. It inverts when the reading happens: instead of storing raw text and searching it when a question arrives, every conversation is read once as it comes in.
Each conversation produces typed signals rather than embeddings: a problem, a request, a question, a piece of praise — each one tied to the person who said it, the moment they said it, the conversation it came from, and the exact quote. Structured records with links, not points in a vector space.
That changes the ceiling. A query-time model's recall is bounded by k, because it decides what matters under a budget, at the moment you ask. AOT decided what mattered at ingestion, exhaustively, with no budget pressure and no question to bias the read. Asking how many customers raised something becomes a count over structured records instead of a similarity search over text, which is why its coverage doesn't move when the question changes.
AOT is proprietary, and your data is never used to train models. Your conversations produce your signals, inside your workspace.
Why our own floor is unfairly low
The 78.9% figure needs explaining, because it is the number a skeptic should attack first. It counts one thing only: whether a signal is anchored to the exact ~2,400-character passage that the answer key drew its quote from. It has nothing to do with matching phrases or wording.
Passage boundaries are arbitrary cuts through a conversation. A customer makes a point across thirty seconds of speech, that lands across a cut, AOT anchors its signal to the half it judged most representative, and the answer key happened to quote the other half. Scored as a miss, even though the pipeline caught the point. So we measured how far away the nearest signal actually is.
Forgive a single chunk boundary and coverage is 88.5%. Forgive two and it is 92.9%. Only 1.0% of the verified evidence sits in a conversation AOT extracted nothing from at all.
We publish 78.9% as the floor anyway, because the honest lower bound is the one that assumes none of those near misses count. The real number is somewhere between 88.5% and 99.0%, and the interesting engineering question the table raises is chunking, not comprehension.
The control: pay a model to read every word
The strongest possible query-time approach is no search at all — put every token in front of a model, per question. We ran it live rather than arguing about it: a scan across all 220,698 passages, then aggregation into an answer, scored by the same blind judge as everything else, on three corpus-wide questions.
Total access to every token bought one extra theme on one question, for about a thousand times the cost and fourteen hours of waiting. The bottleneck was never getting the text in front of a model. It is that reading 101 million tokens with no structure produces broad summaries, while the question asked for specific themes with counts.
What it costs to keep asking
The obvious question about a model that reads everything up front is whether it is worth it. The surprise in our numbers is that per question, it barely costs more than vector search: three cents against two. AOT is not winning on price per query. It is winning on what a query returns.
That reframes the comparison. To reach even 11% of the evidence, query-time search has to pull a hundred passages a question. That is five cents. AOT pulls none, costs three cents, and returns 99%. The only query-time route to the same coverage is reading the whole corpus for every question, at $33.55 a question, forever.
Cost per answer, not cost per query
Price on its own hides the trade. A question is cheap or expensive only next to what it hands back. Line the three approaches up by what one question costs and what that money buys, and the market splits into three.
Query-time search
$0.050
11% of the evidence
One hundred passages a question. Cut to forty passages and you save two cents and fall to 6%.
AOT
$0.030
99% of the evidence
Cheaper than the search that returns 11%, because the reading already happened at ingestion.
Read the whole corpus
$33.55
99% of the evidence
101 million tokens for every question. Same coverage as AOT, 1,100 times the price, fourteen hours instead of a minute.
This is why the argument is not really about price. Query-time search is not expensive, it is small. Paying more for it buys a bigger slice of a small number. AOT does not sit further along that curve. It sits off it, because the expensive part, reading everything and deciding what matters, happened once at ingestion instead of every time somebody asks.
Four ways to attack these numbers
Every one of these is in the paper too. If you are going to poke holes, start here, and we would rather you did.
- The answer key is built from raw text, which is the same material retrieval searches. That shares a lens with the systems being graded. Cross-lab verification narrows it; it does not erase it.
- Our query-time baselines are ordinary production setups — no reranking, no query rewriting, no late-interaction models. A tuned stack beats 11% at 100 passages. The structural argument in Figure 2 is what we think survives a better stack, and we would like to see someone test that.
- AOT coverage is a range, not a point. 78.9% of evidence has a signal on the exact passage; 99.0% comes from a conversation that produced signals. Both numbers are on this page because the honest claim needs both.
- One workspace, one snapshot, 24 themes, 836 pieces of evidence, 95% confidence interval of 98.3 to 99.7 on the headline. Strong for this corpus. Directional for yours until it runs on more.
Check the numbers
Every figure and table on this page comes out of one JSON file, which includes the per-theme breakdown, coverage at every budget, confidence intervals, and the full-scan results. The paper adds the complete method, the related work, and the things that went wrong along the way.
