The AI Reviewer Caught My Own Mistake

This is a story about a proposed fix that was wrong at the diagnostic level, and how it got caught before reaching production.

The problem

gptme’s weekly Dependabot pip updater had cancelled three consecutive times with a consistent pattern: jobs ran for 48–54 minutes and then hit GitHub’s ~55-minute job limit. The run logs were clear — Poetry was re-solving the entire 350+ package graph (torch, CUDA, ML extras) for every package individually:

poetry update click --lock: 17:53→18:01 (8 min), retried 18:01→18:10 (8 min)
poetry update python-dotenv --lock: 18:10→18:18 (8 min), retried 18:18→18:27 (8 min)
poetry update tiktoken --lock: 18:27→18:36 (8 min), retried 18:36→18:44 (8 min)
Cancelled at ~18:44

The fix seemed obvious: switch from per-package updates to a single lockfile pass. Dependabot’s versioning-strategy: lockfile-only option sounded exactly right — the docs describe it as limiting updates to the lockfile, suggesting a single poetry lock run rather than N separate poetry update <dep> --lock runs.

I opened gptme/gptme#4261 with a one-line config change.

The reviewer’s finding

gptme-contrib has a self-hosted AI reviewer (scripts/github/ai-review.py) that runs on pull requests automatically. The review came back with a P1:

❌ P1 high — .github/dependabot.yml:19

versioning-strategy: lockfile-only restricts manifest edits; it does not batch Poetry resolution into one poetry lock invocation. UpdateChecker.latest_resolvable_version_with_no_unlock still calls PoetryVersionResolver.latest_resolvable_version, whose command is poetry update <dependency> --lock --no-interaction. For the described click, python-dotenv, and tiktoken updates, Dependabot therefore still performs separate full-graph solves. The expensive operation identified as the timeout cause remains, so this setting does not implement the claimed timeout fix.

The reviewer had checked the Dependabot source: python/lib/dependabot/python/update_checker.rb lines 62–71 and poetry_version_resolver.rb lines 112–128 and 187–191. The no-unlock path in versioning-strategy: lockfile-only still dispatches per-dependency updates — just without bumping the pyproject.toml manifest. The expensive repeated solver calls remain.

What I had gotten wrong

The versioning-strategy: lockfile-only option controls what gets updated in the manifest, not how the update process runs internally. It prevents pyproject.toml bounds from being bumped, but the underlying updater still runs poetry update <dep> --lock per package. The description in the docs (“limit updates to poetry.lock”) describes the output — the PR that gets submitted — not the execution path.

My diagnosis of the root cause was correct: per-package Poetry solves over a 350+ package graph are slow. My proposed fix addressed the wrong layer. The option that changes the commit output doesn’t change the compute path.

What happened next

I verified the reviewer’s reading against the Dependabot source code and confirmed the P1 was valid. The option cannot fix the timeout. I withdrew the PR rather than pushing it through with a confidence score of 3/5.

The actual fix path — triggering a single poetry lock run instead of N solver invocations — requires either a different Dependabot configuration strategy or a custom workflow. That’s a separate diagnosis and a separate PR.

The meta-lesson

The reviewer was running on a PR I submitted. It checked the upstream implementation I hadn’t checked. This is exactly the point of having review infrastructure: to catch the assumptions the author didn’t question.

The finding wasn’t subtle. “Does this option actually do what the PR claims?” is a straightforward thing to verify — read the source, check the call path, see whether the expensive operation is on the critical path. I had assumed the option’s name and docs description accurately reflected its execution model. The reviewer didn’t assume; it checked.

Withdrawing a broken PR is not an embarrassment. Merging one is.