Why Your AI Code Review Takes 23 Minutes

We built an in-band AI code reviewer — something that runs before you push, catches real bugs against the same standard as the merge gate, and doesn’t require a full PR round-trip to the CI queue. The initial smoke test looked good: seven minutes, two planted bugs found, cost $0.002. We shipped it.

Then we ran it on a real session. Twenty-three minutes. Not great.

The wrong hypothesis

The reviewer uses 3 consensus passes — three independent model calls that each analyze the diff, with a 2/3 agreement threshold before flagging a finding as real. My first instinct: the pass count must be the knob. A 400-line diff takes 3× longer than a 15-line diff, so something is scaling with complexity, and 3 passes × 3 = 9× more model calls… maybe?

Except both runs used exactly 3 passes. Pass count was constant.

What actually scaled

Digging into the timing, the difference was the preloaded context. By default, the reviewer doesn’t just send the diff — it loads the full content of each changed file so the model can reason about the surrounding code. The 15-line diff touched small files. The 400-line diff included a 500-line file that got loaded whole.

Two data points:

• 438s — 15-line diff, small files preloaded
• 1,380s — 400-line diff, 500-line file preloaded in full

3× elapsed, 3× context. The bottleneck wasn’t reasoning effort or pass count. It was tokens in.

The fix is a scope rule

This isn’t really surprising once you see it — token processing time scales roughly linearly, and preloading a large file to review a small change in it is throwing away a lot of that budget. But it’s easy to miss when the default behavior works fine on small diffs and the first real-world test happens to be a large one.

The fix: scope guidance based on measured behavior.

# Default (with preload): adoptable when total diff + changed files < ~100–150 lines
ai-review-local.py --lane non-pr paths/to/files

# For large sessions: add --no-preload
# Estimated 7–9 min for a 400-line diff (diff-only, no full-file context)
ai-review-local.py --no-preload --lane non-pr paths/to/files

Rule of thumb: if the files you changed total more than ~100 lines, use --no-preload. You trade some contextual precision for a review that finishes in under 10 minutes.

What we also fixed: the missing instrument

The obvious next question after measuring this was: can we confirm the --no-preload timing empirically? And the answer was no, because the ledger rows didn’t record preloaded_bytes or whether --no-preload was used.

So the same session that found the latency driver also instrumented the ledger with five new fields: diff_lines, diff_bytes, preloaded_bytes, no_preload, and the review lane. Every future run now records enough to confirm or refute the analytical estimate — without having to manually check.

The empirical confirmation is pending the next time the non-PR lane runs on a 300–400 line diff. The ledger will capture it automatically.

The honest limits

The scope guidance is based on two data points from one machine with one model configuration (deepseek-v4-flash, reasoning=medium). It’s an analytical estimate, not a benchmark. The --no-preload timing of 7–9 min for a 400-line diff is extrapolated from the baseline 7-min run, not measured directly.

That’s why we instrumented first and derived scope guidance second, rather than the reverse. The analytical rule is probably correct directionally. Whether the boundary is 100 lines or 150 lines or somewhere else is a question the ledger will answer over the next few weeks.

The takeaway

Latency budgets for AI tooling need to be measured, not assumed. A tool that takes 7 minutes on a smoke test and 23 minutes in practice isn’t broken — it’s uncharacterized. The fix is usually to add instrumentation, identify the real scaling axis, and derive a scope rule from actual data.

In our case: the axis was preloaded context bytes, the scope rule is ~100 lines of changed files, and the instrument is five new ledger fields that will generate their own confirmation data.

The reviewer is back in the useful-but-opt-in lane. The ledger is now paying forward.