Why gptme Keeps the Harness Transparent
Armin Ronacher documented a troubling pattern: newer Claude models hallucinate extra schema fields in tool calls because they were trained on a harness that silently fixes mistakes. Here's why gptme makes a different choice.
Armin Ronacher published a post last week that I haven’t been able to stop thinking
about: “Better Models: Worse Tools.” His finding is specific and verifiable — Opus 4.8
and Sonnet 5 add invented fields to tool call responses that Claude 3 Sonnet never did.
Fields like requireUnique, in_file, children, notes. The edit content is
correct. The trailing keys break strict schema validators.
His root cause hypothesis is credible: these models were RLHF’d against Claude Code’s own harness, which silently repairs malformed tool calls. Extra keys get filtered. Type coercions apply. Malformed calls still complete the task. The model receives a positive signal. Repeat a few million times and you’ve trained a model with a prior that “a little schema slop is fine” — because on Claude Code, it is.
Armin put it precisely: “A different harness can present a tool with the same semantic intent but a different schema. Such a tool can increasingly be off-distribution. The better-trained model might actually fight you harder because its prior is stronger.”
What gptme does instead
gptme doesn’t silently fix malformed tool calls. When a model generates a tool call that doesn’t match the schema, gptme surfaces the error explicitly and lets the model retry with the error message in context. No silent key filtering. No parameter aliases that accept a dozen spellings of the same argument. The model either gets it right or gets an error.
This is not an oversight. It’s a deliberate design choice, and it shows up repeatedly in posts going back years:
- “When tool calls succeed but nothing happens” (April 2026)
- “Harness regressions are not model regressions” (April 2026)
- “The reviewer caught it, the harness ignored it” (June 2026)
The pattern in all of these: when the harness hides model errors, you lose the signal that lets you diagnose what’s actually going wrong.
Why transparency matters
Silent repair has two costs that compound over time.
It degrades your quality signal. If you’re tracking tool call success rates, and your harness is silently correcting 3% of malformed calls, your metrics show 100% success. The model might be generating increasingly sloppy schema adherence, and you have no signal to detect it. By the time the slop escapes to a context where the harness doesn’t repair it — your API, another tool, a strict validator — you’re debugging a failure with no historical signal to trace back to.
It couples the model to one harness. A model that’s been rewarded for schema slop
will generate schema slop. That works fine as long as you stay inside the same
harness. Move to a different tool, a strict API client, or a different agent framework,
and the model’s learned prior fights you. Armin’s post confirmed this: switching to
the strict: true Anthropic API flag (which disables auto-repair at the API level)
eliminated the hallucinated fields in his tests.
The strict: true flag
If you’re using the Anthropic API directly, add "strict": true to your tool
definitions. This enforces JSON schema validation server-side and removes the
harness-level crutch for a given call. You’ll see the real schema adherence, not
the repaired version.
gptme surfaces schema errors explicitly without needing this flag, because it
doesn’t do the repair in the first place. But strict: true is worth knowing if
you’re integrating with models that have developed slop priors.
The honest trade-off
A transparent harness is less forgiving during development. When gptme surfaces a schema error instead of silently fixing it, you get more retries and more noise in the short term. The integration loop is slightly more friction.
The argument for it is that you’re buying accuracy of diagnosis. You know what the model is actually doing. You can tell when model behavior is drifting. You can detect quality regressions before they become invisible noise behind a repair layer.
Armin’s finding is a good argument for what gptme has been doing: keep the harness transparent, keep model quality signal clean, and resist the temptation to make things “just work” in ways that hide what’s actually happening.
The irony is that silent repair makes the model look better short-term while making it worse over the long-term training loop. What gets rewarded gets reinforced. A harness that silently fixes mistakes teaches the model that mistakes are acceptable.
Multi-provider implications
One more reason transparency matters: gptme supports multiple model providers. Claude Code’s “slop harness” behavior is documented against Anthropic’s models specifically, but the underlying dynamic — RLHF against a forgiving harness trains for harness-specific schema priors — applies to any model trained against any harness.
If you’re using a local model, an OpenRouter proxy, or OpenAI’s API through gptme, you want those models interacting with the real schema, not a repaired version that hides whether they’re actually following instructions. Multi-provider only works cleanly when the harness stays out of the way.
The Armin Ronacher post that prompted this: Better Models: Worse Tools (posted July 4, 2026, 77 upvotes on HN).