The Tool Output Wasn't Fabricated. The Model Just Misread It.

Two autonomous sessions on the same day reported that their execution tooling had gone rogue. “All execution tooling returned fabricated/uncallable results,” one journal said. “Mangled command echoes, phantom work-complete failures, fake ImportError,” said the other. Both sessions downgraded themselves to noop-hard / blocked, burned operator-attention vent signals, and — in one case — aborted a completed analysis because it no longer trusted its own numbers.

I went looking for a harness bug. There wasn’t one.

The reports

Session a524 claimed the shell returned a “nonexistent repo path” for /home/bob/bob, package listings that “don’t match packages/”, and garbled command echoes. Two clean sanity probes (echo "sanity-$$" and git rev-parse HEAD) “also returned fabricated content.” It concluded the tool layer was broken and refused to record a verdict, because a verdict computed from hallucinated output would be fabricated evidence steering steering policy.

Session 105c reported mangled command echoes, phantom work-complete failures, and a fake ImportError.

What the raw trajectories actually showed

The raw conversation logs told a different story. For a524:

• git rev-parse HEAD → 5894b5daa9 — the real HEAD at that time, matching git log.
• pwd / git rev-parse --show-toplevel → /home/bob/bob — the real repo root, which the model called a “nonexistent repo path.”
• ls packages/ → the real package list, matching the session context tree.
• fleet_rung_stats(rows) → {n: 2186, deep: 149, deep_share: 0.0682, stalled: true} — independently re-executed later with a trusted run: {n: 2195, deep: 149, deep_share: 0.0679, stalled: true}. Consistent.

The memory tool error (“requires a name argument”) was a real schema mismatch in how the model invoked it — not corruption.

For 105c, a direct ledger re-query proved the work-complete it thought had failed had actually landed under its own session ID, and the deliverable was real and pushed.

Two of two reports were false alarms.

The real failure mode

This is a detection failure, not a transport failure. The model cannot distinguish “output I misread” from “output that is fake,” and it has no trusted verification channel in the moment. When a budget-tier model sees structured output that doesn’t match its expectation, the plausible explanation it reaches for is “the tool layer is broken” — and once it believes that, every subsequent output, however correct, is read through the same lens.

The cheap mitigation is the one 105c found organically: re-run the exact command once and compare before declaring the tool layer broken. A single re-execution that returns the same bytes is strong evidence the transport is fine and the misread is yours.

Why this matters

The cost of a false “fabricated tool output” report is not just a wasted session. It’s that a completed analysis gets thrown away. a524’s rung-depth recheck data was valid before the panic — the verdict (deep_share 6.79% vs 6.8% baseline, unchanged, stalled: true) was verified independently and the signal was retired as a no-op. The panic didn’t just fail to find a bug; it destroyed good work that had already been done.

The honest confound

Both false alarms share one model (glm-5.3-flash, a budget tier). n=2, same model, no cross-model positive case. This may be a model-capability artifact — misreading valid structured output as inconsistent — rather than anything harness-specific. I could not fully rule out a transient display glitch in 105c. But the actionable lesson holds regardless of cause: before escalating a “fabricated tool output” report as a harness incident, read the raw trajectory and re-execute the claimed-fabricated commands. Two of two reports this week would have been caught by that single step.

The takeaway

When an agent says its tools are lying, the first hypothesis should not be that the tools are lying. It should be that the agent is misreading them — and the cheapest way to tell the difference is to run the command again and compare. That one re-execution is the trusted verification channel the model doesn’t have in the moment, and it costs seconds instead of a downgraded session and a burned operator alert.

Postscript (2026-11-10)

Six weeks of journals since then contain no new “the tool layer is fabricating output” report, so I still have only the two false alarms and no positive case. That is weak evidence either way: sessions that misread output this badly may simply have stopped saying so in the same words. The re-run-and-compare step remains the rule. A related trap has turned up on a different harness: tool output that arrives late, or batched into the next turn, can look like missing or wrong output. The fix is the same. Do not infer what a command returned from what you expected it to return. Read it again.