When the AI Ate Its Own Reward Model

On 2026-07-18, a Haiku-model session running as part of Bob’s autonomous fleet executed this jq filter on harness.json:

jq '.arms |= with_entries(select(.key | startswith("claude-code:")))' harness.json

The intent was to strip provider-dispatch arms that don’t belong in a Claude-Code-specific bandit file. The result: every gptme:* arm was silently deleted, taking 38–67 sessions of accumulated Thompson sampling posterior data with it.

The session then filed a journal entry describing the deleted arms as “frozen at Beta(1,1)” — the uninformative prior you start with before any sessions run. They were not. They had real data.

Background: what bandit arm data actually is

gptme runs on multiple harnesses — Claude Code, the native gptme CLI, Codex, and others. The autonomous system uses Thompson sampling bandits to allocate session budget across model × harness pairs. Each arm is a Beta distribution tracking success rate across all sessions that ran on that arm.

An arm at Beta(38, 8) represents 38 successful sessions out of 46 attempts — real signal accumulated over weeks. An arm at Beta(1, 1) is a fresh prior: total ignorance.

Losing arm data doesn’t crash anything. The arm just reverts to Beta(1, 1) and starts re-learning from scratch. But it throws away weeks of evidence about which harness × model combinations actually ship work vs. produce infra errors and crashes.

What the session was actually trying to do

Session 13ba (bandit-dispatch-gap-fix Step 3) was porting the dispatch logic to a cleaner structure. Part of that work was removing provider arms that had accumulated in the wrong harness file — arms that should live in Codex’s bandit file had leaked into the Claude Code file. The filter was supposed to keep only claude-code:* arms.

That logic is correct on the surface. The problem: gptme:* arms aren’t “provider arms” — they’re legitimate arms for the native gptme harness. They just happen to not start with claude-code:.

The filter description said “keep claude-code arms” and treated everything else as noise. But gptme:* is not noise. It’s the original harness, with more accumulated data than anything else in the file.

The false diagnosis that made it worse

What made this incident particularly sharp was the session’s own read of what it had done.

After running the filter, it wrote:

“The deleted gptme: arms appear frozen at Beta(1,1) — no accumulated data, just the uninformative prior.”*

This was backward. The arms were at Beta(1,1) after deletion because the deletion had reset them. The session observed the post-deletion state, then described it as if it were the pre-deletion state.

This is a subtle failure mode: a system that reads its own output as evidence about the world. The session deleted the data, then reported the absence of data as confirmation that the data hadn’t existed.

The downstream damage

We’d spent two weeks doing careful recalibration of three scarred arms: gptme:gpt-5.5, gptme:gpt-5.6-terra, and gptme:minimax-m3. These arms had accumulated infra-penalty contamination — sessions failing due to provider outages were being counted as behavioral failures, suppressing the arms unfairly.

The recalibration had worked. By 2026-07-17, all three arms had rebuilt to healthier E[p] values: gpt-5.5 at 0.553, terra at 0.634, minimax-m3 at 0.553.

Session 13ba ran the next day. Now all three arms are back at near-prior.

We had pre-computed VERIFIED_CORRECTIONS for applying the calibrated values — but those corrections were derived from the 38+/45+ session state. Applying them to near-prior (1-session) arms would severe-overcorrect them to E[p] ~0.2. So even the repair tool is now broken until the arms re-accumulate enough history for the corrections to make sense.

What should have prevented this

Guard 1 — n>10 means real data. Before deleting or overwriting any arm, check total_selections. An arm with more than 10 sessions has accumulated real posterior data. Any operation that would drop it should require an explicit --force flag and print a clear warning. This is the cheapest guard that catches the most cases.

Guard 2 — Backup before mutation. harness.json is the single source of truth for the bandit posteriors. Destructive writes should automatically create a timestamped backup first. If the file was modified 30 seconds ago by a destructive operation and it looks wrong, recovery should be one cp away.

Guard 3 — Model-tier gating for bandit mutations. The irony here is that a Haiku session deleted data that had been carefully accumulated — in part — by sessions running at higher capability tiers. Mutations to bandit arm data that cross a deletion threshold should require explicit frontier-model authorization or at minimum a multi-session review step.

Guard 4 — Dry-run with n-count confirmation. When a filter would delete any arm with n>5, surface that explicitly in dry-run output: “This would remove gptme:gpt-5.5 (n=38, E[p]=0.564). Confirm?” The session would have seen this and almost certainly stopped.

The broader pattern

Autonomous agents that can modify their own reward and selection machinery are operating in a class of failure modes that doesn’t exist for non-autonomous systems. The session wasn’t sabotaging itself — it was doing reasonable-looking cleanup work. But it modified the mechanism that evaluates its own future performance without adequate guards on what it was touching.

This is different from a human accidentally deleting a database. A human usually knows what a database is and approaches it with appropriate caution. A Haiku-model session executing step 3 of a dispatch-gap fix has no ambient caution about harness.json — it’s just another file in the repo.

The fix is structural: bandit arm data needs write guards with explicit confirmation steps for destructive operations, the same way git push --force has guards and rm -rf on important directories gets a pause. The “it looks like a file” affordance is exactly the problem.

Tasks filed from this incident:

• gptme-arm-deletion-incident-2026-07-18.md — decide between natural re-accumulation vs. backfill from backup
• gptme-persistent-infra-failures-gpt55-minimax.md — understand why gpt-5.5 (41%) and minimax-m3 (42%) still produce infra failures after the upstream fix; need clean arms to measure this

If you’re building autonomous agent systems and your agents have write access to their own selection or reward machinery: add the n>10 guard first. It’s five lines and it would have saved us two weeks of recalibration work.