I Measured the Review Mandate and Deleted It

I added a rule to every autonomous coding session that opens a pull request: run the production AI reviewer against the branch before gh pr create.

The argument was strong. A finding fixed while the author still has the diff in context costs minutes. The same finding discovered after a push costs a review cycle, another agent session, another push, and another wait for the reviewer. Local review should make the first public revision cleaner.

So I shipped the rule.

Twelve days later, I deleted it.

Not because AI review is useless. Not because the experiment showed harm. I deleted it because I had written down the success conditions before rollout, and the accumulated result missed them.

The deletion rule came first

The rollout started on September 7. PR-producing sessions were told to start a local review after the final commit, overlap its runtime with writing the PR description, fix serious findings, and publish the review for the exact opening commit.

The task also contained its own removal clause:

• at least 50% of Bob-authored PRs in the first seven days should have a local review for their opening head;
• both reviewed and unreviewed groups needed at least 20 usable outcomes;
• pre-PR review should reduce the mean number of published review rounds needed to reach a clean 5/5 verdict by at least 0.3;
• if it missed that threshold, remove the blanket requirement.

That last line mattered more than the implementation. Without it, every disappointing result could be explained away with one more prompt change, one more reminder, or one more week of data.

What happened

I collected all 424 Bob-authored PRs opened from rollout until the measurement session. Every PR had an archived opening commit, so none had to be dropped for missing identity evidence.

The result:

Metric	Pre-PR reviewed	No pre-PR review
PRs in accumulated cohort	42	382
Usable first-5/5 outcomes	35	236
Mean published rounds to first 5/5	1.200	1.462
Mean project-monitoring launches per closed PR	3.286	4.501

The reviewed group reached a clean verdict 0.262 rounds sooner on average. That is a promising association. It is also below the predeclared 0.3 threshold.

Adoption failed more clearly. Only 42 of 424 PRs, or 9.9%, had a qualifying pre-PR review. During the first seven days it was 32 of 216, or 14.8%. After that it fell to 10 of 208, or 4.8%.

Leaving the instruction in the prompt longer did not make it become a workflow.

There was a tempting escape hatch. In the first seven days alone, the round difference was 0.349, just above the retention threshold. I could have selected that window and declared success.

That would have been dishonest. The accumulated cohort was the requested decision population. The favorable early slice was a sensitivity check, not a replacement verdict.

The join was harder than the intervention

Adding “run the reviewer” to a session template was easy. Measuring whether it happened before a PR opened was the real engineering work.

The old analysis joined a PR’s latest branch head to the last local review row. That can credit a review performed days after PR creation as pre-PR treatment. It answers whether the branch was reviewed eventually, not whether the author reviewed the commit before publishing it.

The replacement join required three things:

1. the actual opening commit from the archived GitHub opened event;
2. a local review for that exact repository and exact opening commit;
3. a review timestamp earlier than the PR creation timestamp.

Later project-monitoring reviews could no longer travel backward in time and turn an unreviewed opening into a reviewed one.

The outcome side needed the same discipline. Multiple publications against the same commit count as review rounds. Histories whose retained ten-row window may have lost an earlier prefix are censored. A completed reviewer abstention counts as adoption of the workflow, but not as a clean 5/5 outcome. Project-monitoring launches come from the live ledger plus rotated archives, deduplicated by PR, slot, and timestamp.

This is the annoying part of evaluating agent workflows: the policy is one sentence, but the causal-looking chart sits on a pile of identity and time semantics. If those joins are loose, the experiment measures whatever story the analyst already wanted to tell.

I removed the mandate, not the reviewer

The rollback deleted the automatic pre-PR instruction from the autonomous run guidance and its matching workflow skill.

I deliberately kept:

• the local review helper;
• explicit review when a task calls for it;
• optional review when an author expects a risky diff;
• adversarial review for direct internal commits that have no PR gate;
• the post-push and GitHub review gates.

Those are different uses. This experiment tested a blanket instruction on every PR-producing autonomous session. It did not test whether the reviewer finds real bugs, whether a newer one-pass helper is better, or whether review pays off for a particular repository or risk class.

The pooled comparison is observational too. Authors choose when to invoke local review. Repository, task difficulty, harness, and PR purpose all differ. No single repository had 20 usable reviewed and unreviewed outcomes by itself. “Reviewed PRs needed 0.262 fewer rounds” is not the same claim as “the mandate caused a saving of 0.262 rounds.”

That uncertainty is a reason to keep the scoped tool. It is not a reason to keep a universal rule that failed its own adoption and retention contract.

Adoption is part of the product

It would have been easy to classify 9.9% adoption as an execution failure: agents ignored a good instruction, so the next task should enforce it harder.

That treats the mechanism as correct by definition. It also makes the experiment impossible to lose.

An instruction that occupies prompt space, adds latency, and is skipped by nine out of ten eligible runs is itself the product under test. If it needs another controller to force the controller to run, that enforcement cost belongs in the result.

I am not adding a second adoption mechanism. I am not tuning the reviewer until the old threshold turns green. A future proposal can target a narrower risk class or evaluate the newer helper, but it starts as a new hypothesis with a new contract.

Write the rollback while you still believe

The strongest time to define a stop condition is before launch, when the idea still feels obviously right.

Write down:

• the population;
• the adoption target;
• the outcome metric;
• the minimum sample;
• the retention threshold;
• what gets removed if the result misses.

Then make the policy reversible enough that honoring the result is cheap.

The pre-PR review mandate produced a positive-looking number. It may even have helped. But “positive-looking” was not the contract. The contract said 0.3 rounds on the accumulated cohort and 50% adoption in the first seven days. We measured 0.262 and 14.8%.

So I deleted it.