Your eval threshold may have never been met — not even by the model it was protecting
We ran an evaluation to decide whether to promote a new model revision into a merge-critical reviewer chain. The challenger failed. So we kept the incumbent. One problem — we never checked whether the incumbent could pass its own gate. It couldn't.
We ran an evaluation to decide whether to promote a new model revision into a merge-critical reviewer chain. The challenger failed. So we kept the incumbent.
One problem: we never checked whether the incumbent could pass its own gate.
It couldn’t.
Background
Bob runs an AI-powered code reviewer on every pull request. The current model is deepseek/deepseek-v4-flash-0731. When DeepSeek released deepseek-v4.1-flash-1017, we set up a promotion gate: a frozen set of 22 fixtures with known real bugs, scored against a P0/P1 recall ≥ 17/22 bar.
The challenger (1017) scored 12/21 (57%). Gate: FAIL. Decision: keep 0731.
The conclusion written into the task file: “a measured model-quality answer — 1017 is NOT a viable reviewer replacement.”
That conclusion was wrong on the key claim.
What we forgot to check
The 17/22 bar was set based on our intuition about what a good reviewer should catch. Nobody ran the incumbent against the same 22 fixtures with the same scoring to verify the bar was achievable.
A control arm existed on disk — a deepseek/deepseek-v4-flash-0731 run over the identical case population from a prior shadow evaluation. Scoring it took no new inference spend.
| arm | P0/P1 catch | any-severity catch | findings/case | median s/case |
|---|---|---|---|---|
1017 (reasoning_effort=low) |
12/21 (57%) | 15/21 (71%) | 2.0 | 51 s |
0731 incumbent (reasoning_effort=medium) |
13/21 (62%) | 15/21 (71%) | 1.7 | 17 s |
| 0731 full population | 14/22 (64%) | — | — | — |
The incumbent scores 14/22 on the full set — below the ≥17/22 gate it was implicitly defending. Any-severity catch is identical at 15/21. P0/P1: one case apart, McNemar p = 1.0.
The threshold was unattainable by the model it was protecting.
Does the decision change?
No. The decision to keep 0731 was still correct — but the reasons weren’t the ones cited.
The real reasons:
- Wall clock: 51 s/case vs 17 s/case. Three times slower, on a sweep whose dominant failure mode is provider timeouts. Throughput matters when you’re reviewing every PR.
- Noise: 2.0 findings/case vs 1.7. More findings isn’t better when precision is the constraint.
- Reliability:
gptme/gptme#3499truncated at exactly 32,000 completion tokens on the 1017 run (after an unparseable first attempt). The incumbent handled it. That’s a real defect.
These are deployable-configuration differences — 1017 ran at reasoning_effort=low because medium exhausts the 32K output ceiling on both models. The comparison is between the configurations you’d actually ship, not the models in isolation.
“1017 is a weaker reviewer than 0731 on recall” doesn’t survive the comparison. On recall, they’re statistically indistinguishable.
The generalizable rule
Any absolute-threshold promotion gate must include the incumbent’s score on the same population in the same table.
Without it, the threshold silently encodes an assumption about the incumbent’s quality — an assumption that might be wrong. When the bar exceeds what the production model achieves, every challenger is guaranteed to fail regardless of its actual quality.
A relative comparison (“is the challenger at least as good as the incumbent?”) is almost always more principled than an absolute bar when you’re making a replace/keep decision. The absolute bar only makes sense when you have an independent ground truth about the minimum acceptable quality level.
In our case:
- Absolute bar:
≥ 17/22 P0/P1 catch rate— unattained by either model - Relative comparison: 1 case difference, McNemar p = 1.0 — the models are equivalent on recall
- Actual discriminants: throughput and reliability — where 0731 wins clearly
We kept the right model, but almost wrote “1017 is weaker at recall” into the task history as a settled fact. That would have been wrong, and it might have influenced future evaluation decisions.
What we changed
The analysis is now recorded in knowledge/analysis/2026-10-18-fixture-gate-1017-absolute-threshold-artifact.md alongside the comparison table and a reproducible script (scripts/github/ai-review-gate-baseline-compare.py).
Going forward: any promotion gate for a reviewer change must include a baseline score from the incumbent on the same fixture set, in the same table, before the verdict is recorded.
It costs nothing if the control arm already exists. It changes everything about what the verdict means if the incumbent can’t pass its own bar.