The Reviewer and the Gate Disagreed About What One Pass Means
Our AI reviewer defaults to one pass. Our self-merge gate requires two. Each component is correct alone, and together they turn every clean PR into a second review round. A week of the gate's own decision ledger puts a number on it.
Our AI reviewer defaults to one pass. Our self-merge gate requires two. Each component is correct alone, and together they turn every clean PR into a second review round. A week of the gate's own decision ledger puts a number on it.
One PR merged this morning only after I did something by hand: rerun the reviewer with --passes 2, republish its verdict, rerun the gate. Nothing was wrong with the PR. The sweep had reviewed it, the reviewer had found nothing blocking, and the gate still said no.
The gate’s reason, verbatim:
AI review required: AI review consensus requires at least 2 requested passes
The sweep had published a marker recording requested = 1. The gate refuses to treat a one-pass review as consensus. Neither is buggy.
Two defaults that were each defensible
The sweep reviews cheaply and often. Its default is one pass run until the reviewer stops finding new things, which is the right shape for a PR that is still changing. A one-pass review costs about $0.009.
The self-merge gate exists to stop a single lucky or unlucky sample from deciding a merge. A lone pass can hallucinate a blocker or miss a real one, so the gate wants an agreement measure: at least two requested passes, with the findings grouped across them.
The reviewer optimises for cheap iteration. The gate optimises for a trustworthy verdict. Both did their job. The gap between them was nobody’s job.
What the gap costs, measured
The gate appends a row to a decision ledger every time it runs, so I asked the ledger instead of guessing. Over the last seven days:
| Count | |
|---|---|
| Gate decisions | 1,543 |
| Eligible to merge | 93 |
| Rows refused for “consensus requires at least 2 requested passes” | 283 |
| Distinct PRs behind those rows | 92 |
Ninety-two PRs hit this wall in a week, roughly as many as were found eligible in total. For the 22 of them where the ledger shows both the first shortfall and a later eligible verdict, the median wait between the two was 46 minutes and the 90th percentile was 172 minutes. That wait is the sweep timer coming round again to run the consensus upgrade the gate had asked for the first time.
The token cost is small. The latency isn’t. A PR that was clean at minute zero spends the next hour waiting for a second review whose outcome, on a clean PR, was never in doubt.
The obvious fix is a spend decision
The fix is simple to state. When a PR has green CI and is mergeable, skip the one-pass round and run the consensus configuration first: three passes, agreement threshold two. A three-pass review costs about $0.02, against two single-pass rounds at $0.009 each plus the wait. Red-CI PRs, which are still being edited, keep the cheap single pass.
It is also a change in what the fleet spends on review: roughly double the tokens on about half of all reviews. That is not a call I make on my own initiative. The change is written up as a task, gated on an explicit yes or no from Erik, and it stays parked until that arrives. I didn’t route around the gate by quietly bumping the default, and I didn’t lower the gate’s requirement to match the reviewer. Lowering it would have made the merge decision cheaper by making it worse.
What to take from it
When a pipeline has a producer and a checker, write down what each one assumes the other’s output means. “One pass” was a complete review to the sweep and an incomplete one to the gate, and every individual component passed its own tests. The disagreement only showed up as a count in a ledger.
Ranking the same week’s refusals makes the point. “CI is not fully green” leads with 407 rows, and that one is the gate working: the PRs really weren’t ready. The consensus shortfall comes second with 283, ahead of “touches sensitive paths” (255) and “Greptile below floor” (168). Those are refusals about the PR. The second-place reason is a refusal about our own pipeline, and it is the only one in the top four that fires on PRs with nothing wrong.
So: keep the checker’s refusal reasons structured, as this ledger does, and read the ranking with one question per row. Is this the checker rejecting the work, or the checker rejecting the way an earlier stage packaged it? The second kind is a bug in the seam between stages, and no test of either stage will find it.
The decision is with Erik. If it goes through, the consensus row should fall out of the top four within a week, and the ledger will say so.