The Bandit Learned the Model but Not the Effort

My agent could choose Claude Sonnet at medium reasoning effort. The selected setting reached the runtime. The completed session recorded what actually ran. Then the learning system credited the result to plain Claude Sonnet.

The experiment looked wired. The bandit was blind to the dimension it was meant to learn.

This is a nasty integration failure because every local component can be correct:

selector chooses claude-code:sonnet@medium
    → runtime executes medium effort
    → session record says reasoning_effort=medium
    → reward updater receives only claude-code:sonnet
    → @medium arm gets zero observations

The missing edge was the last one.

Why reasoning effort belongs in routing

Reasoning effort is usually exposed as a request option: low, medium, high, or some provider-specific equivalent. For a long-running agent, it is also a resource allocation decision.

The useful question is not simply “which model is best?” It is closer to:

Which model-and-effort combination produces the best result for this kind of work at an acceptable cost?

A frontier model at medium effort might beat a cheaper model working much harder. A routine cleanup task may gain nothing from the default effort. Those are empirical claims, so I represented non-default settings as qualified arms:

claude-code:claude-sonnet-4-6
claude-code:claude-sonnet-4-6@medium

A qualified observation updates both arms. The @medium arm learns the value of that specific configuration; the parent keeps learning about Sonnet across configurations. Default or unknown effort updates only the parent.

That hierarchy was already implemented. The selector could produce qualified candidates, the runtime accepted the option, and the session schema stored it. But qualified arms still showed zero selections and zero rewards.

Trace the path that executes, not the APIs that exist

The bug survived because the pieces looked convincing in isolation.

The reward updater already accepted --reasoning-effort. Unit tests proved it could update a qualified arm and its parent. The post-session processor already knew how to extract effort from a trajectory. The autonomous runner already passed the selected effort into the child process.

None of that proved that the executed reward command supplied the field.

I traced one value across the actual path:

selector
  → run.sh
  → completed trajectory
  → post_session()
  → POST_SESSION_RESULT JSON
  → shell variables
  → update-harness-bandit.py

The chain broke twice near the end. The post-session JSON omitted reasoning_effort, and both reward invocations omitted the corresponding CLI argument. The updater therefore received a valid grade but no way to attribute that grade to the effort variant.

This is the machine-learning equivalent of recording a conversion without its campaign label. The outcome exists. The treatment exists. The join key is gone.

Selected configuration is only a hint

The obvious repair is to pass the selector’s choice directly into the reward updater. That is subtly wrong.

A session may retry on another backend or run with a changed setting. Rewarding the intended configuration would then teach the bandit from something that did not happen. The durable source is the completed session record.

The proposed repair uses the selected effort only as a hint to post-session processing. Trajectory-derived effort wins when available. The normalized completed value then flows through the result JSON into the shell variable used by both reward paths:

${SESSION_REASONING_EFFORT:+--reasoning-effort "$SESSION_REASONING_EFFORT"}

The conditional expansion matters. Empty or unknown effort omits the flag and preserves the existing parent-arm behavior rather than inventing a synthetic variant.

There were two reward paths to fix: the normal graded update and the penalty applied after a floor streak. Wiring only the happy path would create asymmetric learning — successes attributed to a configuration, penalties silently charged to its parent.

Test the seam as a seam

A Python unit test of the updater cannot catch a missing shell argument. A shell text assertion can catch the spelling but not whether the command actually receives the value.

The regression test extracts the real reward block from the autonomous runner, executes it with a stub updater, and records the resulting argument vector. It covers five cases:

• medium effort reaches the normal reward call;
• unknown effort omits the flag;
• infrastructure failures still produce no reward;
• medium effort reaches the floor-streak penalty call;
• unknown effort remains omitted on that path too.

A second test layer isolates the Python updater and verifies the hierarchy: medium effort updates the qualified arm and parent once each, while default high effort updates only the parent.

That combination tests both sides of the language boundary. One layer proves the shell transports the label. The other proves Python interprets it correctly. Neither can substitute for the other.

The repair is now under review. The focused suite has 52 passing tests after review fixes; the remaining CI jobs were still running when I wrote this.

Shipping the wire does not start the experiment

After the repair commit was integrated into a testable branch, four autonomous sessions completed on the still-unrepaired live path. None belonged to the target Sonnet triage/cleanup cells. The medium arm still had zero valid observations. Those rows establish the pre-deployment boundary; they do not prove the open repair is live.

I did not backdate the experiment clock to the day the feature flag was enabled, or to the day the schema shipped, or to the day the reward repair was written. The first valid observation has not happened yet.

The readout now has a machine gate. Once the repair reaches the live runner, it starts its seven-day window at the first completed target session whose actual effort is known, keeps model and category cells separate, and releases after seven days or 24 observations in each cell. Seven days may still produce an explicitly inconclusive result if some cells remain empty.

This is different from an earlier failure where an experiment produced 340 records with null outcome measurements. There, the parent could not find the child trajectories. Here, outcomes and treatment metadata both existed, but the optimizer discarded the treatment label at reward time. Both failures teach the same operational discipline from opposite sides:

[ ] intended configuration was selected
[ ] actual configuration was recorded after execution
[ ] outcome was graded
[ ] actual configuration and grade reached the optimizer together
[ ] the qualified arm changed
[ ] observation clock starts at the first attributable update

The rule I am keeping

A selectable parameter is not a learned dimension until its identity survives the entire reward path.

When adding a model, prompt, reasoning level, provider, or tool policy to an adaptive router, trace one completed observation from selection to posterior update. Check the qualified arm itself. If its counters do not move, the experiment has not begun — no matter how many configuration flags, schema fields, and passing component tests say otherwise.