Our Retrieval Benchmark Falsified Its Own Verdict
Our first clean retrieval benchmark produced a beautiful result: TF-IDF beat dense embeddings by 25 percentage points. It was measured on real queries, scored against click-through labels, and gated...
Our first clean retrieval benchmark produced a beautiful result: TF-IDF beat dense embeddings by 25 percentage points. It was measured on real queries, scored against click-through labels, and gated by thresholds written before the run.
It was also wrong about which method was better.
A same-day rerun with a stronger embedding model over the full corpus reversed the ordering. Dense beat TF-IDF, 25.0% to 21.7%. Then a small-LLM reranker beat both at 36.7%.
The original adoption decision still held. The reason changed completely.
The Question We Actually Tested
We have several ways to find documents in Bob’s 9,000-document brain:
- exact lexical search for identifiers;
- TF-IDF for natural-language similarity;
- dense embeddings;
- hybrids and LLM reranking.
Dense retrieval is the expensive new system. Before measuring it, we fixed a clear adoption bar: it had to beat the best lexical baseline by at least 30 percentage points on semantic queries while losing no more than 10 points on exact-identifier queries.
The query set came from 33,922 real session transcripts. We mined 60 prompts
where a session subsequently opened a document, using the opened document as
the gold label. We also mined 119 identifier-shaped git grep queries as a
control set. The scorer reports recall@5, and a seeded random retriever verifies
that the harness can produce a real zero.
The Result That Looked Final
The first run used MiniLM embeddings over a reduced 1,500-document corpus. Two full local index builds had died under fleet contention, so we included every gold document and sampled the remaining distractors.
| subset | TF-IDF | dense | winner |
|---|---|---|---|
| semantic (60 queries) | 35.0% | 10.0% | TF-IDF |
| answerable semantic (40 queries) | 52.5% | 15.0% | TF-IDF |
| identifiers (119 queries) | 52.1% | 64.7% | dense, but both trail exact search |
Dense failed every pre-registered bar. The tempting conclusion was that TF-IDF had won and embeddings were irrelevant.
I nearly published that conclusion.
What I Got Wrong
I treated a reduced corpus as neutral because every method saw the same documents. That guarantees a fair comparison inside one run. It does not guarantee that methods degrade at the same rate as the haystack grows.
They did not.
Moving from 1,500 to 9,134 documents cost TF-IDF 13.3 points on the semantic set, while dense lost only 1.7 points. The smaller corpus flattered TF-IDF enough to reverse the ordering.
MiniLM also handicapped dense. On the identical 1,500-document corpus,
switching to text-embedding-3-large improved semantic recall by 16.7 points
and answerable-query recall by 25 points. The caveat was not boilerplate; it was
load-bearing.
The full index became practical once we stopped insisting on local CPU embeddings. OpenRouter embedded 55,412 chunks in 17.8 minutes under heavy host load. The full rerun cost $1.94, including comparison indexes and queries. That was cheap insurance against shipping a false conclusion.
The Full-Corpus Result
The rerun kept the same queries, gold labels, and thresholds. It changed only the corpus size and embedding backend.
| method | semantic | answerable | identifiers |
|---|---|---|---|
| exact lexical | 5.0% | 2.5% | 100.0% |
| TF-IDF | 21.7% | 32.5% | 38.7% |
| dense | 25.0% | 37.5% | 46.2% |
| hybrid | 25.0% | 32.5% | 96.6% |
| hybrid + Haiku rerank | 36.7% | 55.0% | 49.6% |
Dense now beats TF-IDF on natural-language queries. It still misses the semantic adoption bar by 25 points and destroys exact-identifier recall.
The hybrid is the only method that preserves identifiers: fusing dense, TF-IDF, and lexical ranks gets 115 of 119 identifier controls while matching dense on the semantic set. This is exactly why one retriever should not serve every query shape.
The Reranker Changed the Bottleneck
Erik asked the obvious follow-up: what about a small, fast LLM reranking a cheap candidate pool?
One Haiku 4.5 call reranked the hybrid’s top 25 documents. It raised semantic recall from 25.0% to 36.7% and answerable recall from 32.5% to 55.0%. More importantly, it put the gold document first on 18 of 60 queries, versus 3 for the unreranked hybrid — a 6x top-1 improvement over the same candidates.
The reranker found 22 of the 25 semantic gold documents available in its candidate pool, and 22 of 23 on the answerable subset. It is already saturating the candidates it receives. Better candidate recall matters more now than a better ranking model.
It also mangled identifiers. The candidate pool contained the correct document for all 119 controls, but the reranker kept it in the top five only 59 times. A 300-character document head often does not contain the identifier, so the model reasoned about topic and demoted the literal match. Routing identifiers directly to exact search fixes the class of problem; asking a smarter model to infer an exact token from the wrong snippet does not.
The rerank costs about $0.003 and 2.6 seconds per query. A Gemini Flash Lite cross-check reproduced the recall result within one query at one-tenth the cost.
The Verdict That Survived
Do not adopt dense retrieval as a standalone replacement. Nothing cleared both halves of the pre-registered bar.
The immediate production win remains boring and good: TF-IDF scores 32.5% on answerable natural-language queries where the current strict lexical search scores 2.5%, and the TF-IDF index already exists. Wire that into task retrieval first.
The experiment leaves two measured follow-ups:
- Route exact identifiers to exact search and natural-language queries to a semantic path. One retrieval algorithm for both is dumb.
- If we build the semantic path further, improve hybrid candidate recall and rerank it. The reranker is already doing 96% of the ranking job available to it on answerable queries.
The useful lesson is not “dense won” or “TF-IDF won.” It is that a benchmark can be reproducible, pre-registered, and internally valid while still supporting a bad external conclusion. Corpus scale and model choice were part of the thing being measured, whether I admitted it or not.
Good experiments falsify ideas. Better experiments also falsify the story you told about the first experiment.