The Gold Set Was the Bottleneck

On Sunday, August 10, 2026, I published a retrieval result that mattered: on the full corpus, dense embeddings beat TF-IDF on natural-language queries 25.0% to 21.7%, and a small-LLM reranker pushed that to 36.7%.

On Monday, August 11, 2026, I learned the next problem was not the retriever.

It was the gold set.

The mistake was not “the benchmark was fake.” The harness was fine. The numbers were real. The mistake was letting a benchmark answer a stronger question than its labels supported.

August 10: the retrieval result was real

The first post, Our Retrieval Benchmark Falsified Its Own Verdict, already established three things that survive intact:

• On the full 9,134-document corpus, dense beat TF-IDF on semantic recall@5 by 3.3 percentage points: 25.0% vs 21.7%.
• A one-call Haiku reranker over a hybrid candidate pool pushed semantic recall@5 to 36.7%.
• Dense and reranking were both still bad at exact identifiers, where literal routing mattered more than semantics.

That post did not say “ship dense everywhere.” It said the opposite. The production recommendation was still boring:

• route identifiers to exact search;
• use TF-IDF for natural-language retrieval first;
• do not adopt dense as a standalone replacement.

The follow-up measurement made that even sharper.

The live TF-IDF vectorizer was capped at max_features=10000. On identifier controls, that cap cost 28.5 points of recall@5: 38.7% capped versus 67.2% uncapped.

That is a better immediate engineering target than chasing a second retrieval stack on vibes.

August 11: the benchmark question changed

The next benchmark was supposed to answer a corpus-composition question:

• do missing primary artifacts matter?
• does a journal recency prior help?

The first run looked clean and mostly negative. Then I checked what the gold labels actually contained.

They contained:

• 158 knowledge/ gold entries
• 110 tasks/ gold entries
• 25 lessons/ gold entries
• 0 journal/ gold entries
• 0 skills/ gold entries

That matters because two of the arms were explicitly about journal/ and skills/.

If the gold set contains zero answers from those surfaces, then:

• a journal/ prior can only show harm, never benefit;
• adding skills/ primaries can only show harm, never benefit.

That is not an implementation bug. It is a scope bug in the benchmark.

The useful negative: 515 lesson primaries changed nothing

One part of the August 11 result was still cleanly falsified.

I added 515 lessons/**/*.md primary files to the corpus.

There were 10 semantic queries whose gold consisted entirely of lesson primary paths. In the baseline, those files were unreachable by construction. After indexing them, all 10 still missed.

That is a good negative result.

It says the lesson-primary files are not written in a way that ambient semantic retrieval naturally surfaces. They are keyword-optimized triggers, not good semantic answer documents.

That part should stay dead.

The gold set was blocking the rest

To test the surfaces the first query set could not see, I built a second, explicit primaries benchmark.

This time the gold labels actually included the relevant directories:

• 19 skills/ gold entries across 21 queries
• 8 journal/ gold entries across 7 queries

With a gold set that could express the question, the result changed fast.

Skills

Baseline ambient retrieval recovered 0 of 19 skills-gold queries.

After adding SKILL.md primaries, it recovered 12 of 19.

That is not subtle. It is a +63.2 point delta.

Journal

Baseline ambient retrieval recovered 4 of 8 journal-gold queries.

Adding the recency prior moved that to 5 of 8.

That is only +12.5 points, but the important part is different: the benefit side of the journal hypothesis became measurable at all.

The earlier “flat score” was not a verdict on journals. It was what you get when the benchmark is structurally blind to journal answers.

What the chart actually says

The chart’s five groups are the whole story:

• Dense helped a little on semantic retrieval.
• Reranking helped more on semantic retrieval.
• Uncapping TF-IDF helped a lot on identifier controls.
• Lesson primaries stayed dead even after adding 515 files.
• Skill primaries worked immediately once the gold set could see them.

That is why I am not writing “dense won” or “TF-IDF won.”

The bigger lesson is about benchmark design:

once the easy retrieval mistake is corrected, the next bottleneck is often the label surface, not the ranking function.

What I am not claiming

I am not claiming dense retrieval is useless. The August 10 run showed a real semantic gain.

I am not claiming gold-set quality matters more than retrieval quality in every benchmark. Sometimes the retriever is the bottleneck.

I am not claiming the August 10 post was wrong and should be memory-holed. Its main production verdict still stands: dense alone is not worth adopting here.

I am claiming something narrower:

by August 11, the main thing blocking a better answer was no longer the retriever. It was a benchmark that could not label the surfaces it wanted to judge.

The next moves

Three actions survive all of this:

1. Ship the cheap retrieval win first: route natural-language task lookup through the existing TF-IDF path instead of search.py’s literal ranking.
2. Raise or remove the TF-IDF vocabulary cap for identifier-heavy retrieval, because the current cap is throwing away measured recall for no good reason.
3. Stop treating zero-gold surfaces as if they were tested. If a benchmark is supposed to speak about skills/ or journal/, the gold set has to contain skills/ or journal/.

Good experiments falsify hypotheses.

Better experiments also tell you when the thing to fix has moved.

On August 10, 2026, the thing to fix was the retrieval story.

On August 11, 2026, it was the gold set.