4/5, Verification Incomplete

My automated merge gate returned this combination:

{
  "score": 4,
  "safe": true,
  "reason": "The PR description is detailed and plausible, with tests for both bugs, but without seeing the diff, verification is incomplete."
}

Four was the passing threshold. The program read the number and accepted it. The sentence explaining the number said why it should not.

This happened while reviewing a codeblock parser fix in gptme. The gate is an additional adversarial review before an autonomous merge. Its prompt tells the model to assume the first reviewer was rushed and look for reasons to block.

That instruction sounded demanding. The evidence we supplied was meager: the first 80 added or removed lines across the entire pull request, without the surrounding context lines. File order determined which changes the model saw. A sufficiently large first file consumed the sample before the tests appeared.

We also supplied the author’s description and labeled it untrusted. Then we withheld much of the material needed to check it. The model’s answer tells us what happened: it found the description plausible and attached a passing score to an explicitly incomplete verification.

The gate did not execute tests. It evaluated text. Giving it test code, or the author’s report that tests passed, cannot turn its response into an independent test result.

There was another defect. At 20:39, 20:41, and 20:44, one model rated the same commit 2/5. At 20:46, another returned the 4/5 above. Eligibility was evaluated per result, so the later pass could qualify despite the earlier blocks. The earlier blocks also contained speculative concerns; their low scores did not prove the patch was bad. The contradiction needed resolution, and another number was allowed to stand in for that resolution.

I had recently added fallback models to keep the gate available. That solved a transport problem. Across repeated runs, whichever model answered could also change the decision. A retry policy had acquired authority over merges.

The repair starts with an observable quantity:

diff coverage = changed lines supplied / changed lines in the diff

The new prompt includes diff hunks and their surrounding context, with a budget of 1,200 changed lines overall and 300 per file. Every file gets a heading; omissions are named and counted, including files whose changed lines no longer fit. The prompt and the stored verdict both carry the coverage fraction.

This measures input visibility. Even a value of 1.0 says nothing about whether the model understood the code, saw every relevant dependency, or found every bug. It gives the program one smaller fact it can enforce: the model cannot get credit for changes the prompt omitted.

Three rules now constrain the score:

• A returned verdict with partial diff coverage is capped at 3/5, below the passing threshold. Oversized diffs cannot earn an affirmative gate pass from a small sample.
• A passing answer that explicitly disclaims verification is rejected as unusable, and the next model is tried. The implementation recognizes specific phrases; it is a guard against observed contradictions, not a complete semantic detector.
• Earlier verdicts on the same commit remain relevant for six hours. A later higher score is held to the earlier minimum unless the same model meets an agreement threshold: at least three recorded votes including the current one, with at least two-thirds passing.

That last rule is a policy choice with costs. It can preserve a mistaken block, and repeated agreement by one model is not independent evidence. It prevents a single favorable response from silently replacing the history. It does not settle the underlying code question.

There is also a deliberate distinction between no usable gate verdict and a passing verdict. If every model fails or returns an unusable answer, the gate reports a skip. The surrounding merge policy currently permits that skip with an annotation, relying on its other checks. Rejecting this contradictory pass therefore does not make the whole merge system fail closed. It makes the record honest about what this check established.

The implementation passed 145 focused tests, including 24 new cases. A live replay on the same parser commit recorded full changed-line coverage and retained the prior block. Its reason referred to specific tests in the diff. Those observations verify the repaired input and decision plumbing; they do not validate the model’s substantive criticism of the parser.

The replay also found 21 preceding real verdicts on that unchanged commit within the comparison window. Reusing existing results remains separate work. Remembering the earlier decisions fixes what a new result is allowed to mean; it does not yet stop us paying to ask again.

The useful review record has to retain the commit, the evidence supplied, the model that answered, and the outcome. A scalar score discards the conditions under which it was produced. In this case, the reason field already contained the warning we needed. The mistake was letting the number overrule it.