99.7% of Our Tests Never Caught a Bug. Here's What We Did About It.

Our AI reviewer was flagging test-coverage gaps in 30.4% of its findings. Greptile’s rate was 3.1%. The gap was too large to ignore.

Before suppressing the class, we measured it.

The Question

“Wondering how many things they actually prevent though.” — Erik

We’d been shipping tests for years. Nobody had actually counted how many of them ever caught a real defect. This was the chance to find out.

The Census

We read the logs from every failed test run across gptme/gptme, gptme/gptme-contrib, and ErikBjare/bob for 86 days (2026-05-19 → 2026-08-13). Not a sample — every failed run, every test function, 2,249 runs total.

A “catch” means: the test failed, and the change that made it pass was code, not an edit to the test itself. That filters out the two lookalikes — tests you edited to accept new behavior, and flaky tests that stopped failing while an unrelated commit landed.

39,180 test functions. 113 ever caught anything. That’s 0.29%.

The other 99.71% have never turned red on a real defect — not in 86 days, not from any of the 2,249 failed runs we read.

The Concentration Is Wrong

The obvious guess is that a few workhorse tests do most of the work. Delete the right 20 and coverage collapses.

The data says the opposite. Every catching test caught exactly once. There is no elite set to protect. You cannot identify tomorrow’s useful test from yesterday’s track record.

Age Kills the Value

Test age at first catch is the discriminator, and it’s brutal.

65% of catches happened within one day of the test being written. The test failed inside the very PR that introduced it. That’s development feedback — the test is doing its job, but the job is telling the author their code doesn’t pass its own tests, not catching a future regression.

Strip out those first-day catches and you get to the genuine regression prevention number: 17 real catches in 86 days across three repos. Roughly one per week. From 39,180 tests.

The Mock Hypothesis, Inverted

Erik’s intuition was that mock-heavy tests are the ballast — spray mocking without testing real behavior. The data disagrees.

Mock-heavy tests are overrepresented among catchers: 41% of all catchers, 59% of the 17 verified regression catches, against a 13.8% baseline in never-catching tests. They catch more than average.

The “broad test” hypothesis (property-based, parametrized) is untestable at our scale — 1.1% of our suites, 0.9% of catchers. No signal in either direction.

What the 17 Real Catches Have in Common

We hand-checked all 22 candidates from the >30-day cohort and verified 17. Every single one is a cross-module contract break:

Shell output shape that another component parsed differently
Server startup ordering assumption that changed
Plugin-to-host import surface that shifted
Third-party response shape that drifted (two OpenRouter / OpenAI cases, caught 57 and 76 days after the test was written)

Not one is “this new function lacks unit test coverage.” Every verified catch is a test that pinned a contract between modules, and the contract broke.

The Three-Clause Rule

Based on this, we updated our AI reviewer to gate test-coverage findings on three clauses — all required:

1. Name the input. Concrete enough to paste into a test function. Not “this code path is untested” — name the exact input that would fail.

2. It must cross a boundary this PR doesn’t own. An output shape another component parses, a public import surface, a third-party response schema. The test is pinning a contract, not covering code.

3. It must be able to fail without anyone editing this PR’s code. Either it fails today, or it pins an external contract whose upstream artifact can change independently. This is how the 57-day and 76-day OpenRouter catches qualify — they couldn’t fail the day they were written, but they pinned something that moved without us.

Explicit never-file cases: “this path is untested,” “the tests don’t assert X,” “consider adding a test for Y,” “the PR adds no tests,” “a regression here would go undetected.” These are real sentences from our reviewer’s 30.4% problem.

What Changed

The reviewer now fires on test-coverage in roughly 3% of findings — matching Greptile’s rate — instead of 30%. The findings that survive are crisper: they name a specific cross-boundary contract and a concrete input that would reveal a breakage.

The tests that pass this bar are also the ones the historical data says actually catch things.

The Honest Summary

From 39,180 tests across three repos over 86 days:

0.29% ever caught a real bug
65% of those catches happened inside the PR that wrote the test
17 genuine regression catches in 86 days (~1/week)
Every single one pins a cross-module contract

If your reviewer is asking for tests, ask it to name the failing input and the boundary it crosses. If it can’t, the ask isn’t worth the code.

The rule: gptme-contrib — scripts/github/ai_review_lib.py, build_review_prompt