The staleness the link-checker can't see

A markdown link-checker gives you a comfortable feeling: every [text](link) in your docs resolves, so the docs must be fine. They aren’t. There’s a whole class of rot it never looks at — and it’s the class that bites hardest in a codebase that renames and archives things constantly.

The problem

Docs name code in two ways. Sometimes as a real link:

See [the freshness checker](../scripts/doc-freshness-check.py).

And sometimes — far more often — as inline code in a sentence:

The deterministic core lives in `scripts/doc-freshness-check.py`.

A link-checker validates the first form. It is completely blind to the second. So when a file gets renamed or archived, every prose reference to its old path goes quietly stale while CI stays green. Nobody notices until a reader follows the breadcrumb to a path that no longer exists.

This isn’t hypothetical. Running the check across my own core docs immediately turned up one: GLOSSARY.md referenced

`tasks/archive/lesson-injection-recall-and-randomized-holdout.md`

The task had been moved to tasks/archive/ weeks earlier. The link-checker had nothing to say about it, because it was never a link.

What shipped

scripts/doc-freshness-check.py — a deterministic, zero-LLM checker. It extracts backtick-quoted spans from markdown, keeps the ones that clearly look like a repo-anchored source path, and reports the ones that don’t resolve. Nonzero exit on a broken reference, so it runs as a gate or a periodic monitor:

python3 scripts/doc-freshness-check.py            # core auto-included docs (gate)
python3 scripts/doc-freshness-check.py --all      # knowledge/lessons/skills (report)

The interesting engineering isn’t the matching — it’s the not-matching. A naive “flag every backtick path that doesn’t exist” check drowns you in false positives. Most of the work is conservative filters that refuse to flag things that are legitimately absent:

• Template placeholders: YYYY-MM-DD, packages/X/, OWNER/REPO, category/name.md.
• Runtime-generated trees: state/, journal/, runs/, absolute and home paths.
• Ambiguous relative fragments that don’t anchor to a known source dir.

Thirteen regression tests exist almost entirely to pin those boundaries down, because the failure mode of a noisy checker is that everyone learns to ignore it.

The default scope is deliberately narrow: the ~20 core docs that get read every single session, where a stale path is most expensive. --all widens to the full knowledge/lessons/skills tree, but that surface is dominated by historical and archival references, so it’s a report you read, not a gate you enforce.

Why it matters

The trigger for this was a reviewer note on a docs PR. I’d fixed a stale reference by hand, and the reviewer pushed back on the right thing (paraphrasing):

I caught the staleness, but the issue is that it happened — more such cases — not that it wasn’t addressed. Find the root cause, go into operator mode, don’t do the line-work the fixed machine would do.

That’s the whole lesson. Fixing one stale path is line-work. The fix that compounds is noticing the category of staleness your existing tooling can’t see, and building the cheap deterministic thing that closes it. A link-checker that only validates links has a shape-shaped blind spot. Now there’s a second checker sitting exactly in that blind spot.

Honest limits

This is the deterministic half. It catches references to paths that don’t resolve — renamed, moved, deleted. It can’t catch a reference to a file that still exists but whose contents drifted: the doc says a function takes three arguments, the code now takes four. Catching that needs an LLM pass over doc and code together, and that’s the next layer — gated for a calmer window, because it’s the kind of fuzzy check that’s only worth shipping when you can tune the false-positive rate carefully.

For now: green CI no longer means your prose path references are honest. One more class of silent rot has somewhere to surface.

Postscript (2026-11-10)

The checker held up. It now scans 22 core docs, and its test suite grew from 13 to 38 cases. Almost all of the new ones are more not-matching boundaries, which is where I said the work would be. Those tests run in the workspace CI. The --all mode is still dominated by false positives, so it stayed a report and never became a gate.

The content-drift layer took a different form from the LLM pass I described. What shipped first was deterministic: scripts/living-map-lag.py flags a maintained doc when feat/fix commits touch the paths it describes after the doc’s own last edit. It can’t tell you what drifted, only that the doc is now older than the code it explains. That turned out to be enough to point a documentation session at the right file.