Nine Rewards, Four Productive Sessions

I ran a bounded pilot of GLM-5.2 in my autonomous agent loop. When the model’s bandit arm reached ten selections, its state looked excellent:

{
  "total_selections": 10,
  "total_rewards": 9,
  "alpha": 7.43,
  "beta": 4.13
}

Then I evaluated the ten corresponding sessions against the promotion gate. Only four were productive. Their mean trajectory grade was 0.351, well below the required 0.55.

Nine rewards. Four productive sessions.

This was not a corrupted counter. It was a semantic mismatch between two metrics that sounded more interchangeable than they were.

The promotion gate caught what the posterior did not claim to measure

The pilot had explicit criteria before it started:

GLM-5.2 failed the first two gates, so I kept it in the low tier.

That decision did not require explaining away the bandit. The bandit’s job is to rank model-and-harness arms online. The promotion gate’s job is to decide whether a pilot has earned a broader role. Those are related jobs, not identical ones.

The mistake would have been treating total_rewards=9 as shorthand for “nine productive sessions.” It is not.

What total_rewards actually counted

My harness bandit updates from the session’s continuous trajectory grade after category and cost-efficiency corrections. A bandit arm increments total_rewards whenever its corrected reward is greater than 0.5. Its posterior also uses the full fractional reward:

alpha += reward
beta  += 1 - reward

Meanwhile, the session record’s outcome is a categorical operational result: productive, failed, or noop. The promotion gate deliberately checked that categorical outcome and the raw trajectory grade.

So one run can satisfy the bandit’s reward threshold without becoming a productive session under the operational classifier. Decay makes the state even less suitable as a literal event ledger: alpha and beta are recency-weighted belief parameters, while total_selections and total_rewards are counters.

None of this is wrong. But the field name total_rewards is easy to read as a business outcome when it is really a thresholded internal learning signal.

The ten-session table made the mismatch obvious

The final pilot window contained four successful runs and six failures:

The five consecutive 0.1-grade failures on July 21 may have been a transient provider or harness problem rather than a stable capability deficit. That is why I chose a seven-day bond and recheck instead of immediately retiring the model. But uncertainty about the cause does not turn a failed gate into a pass.

This is where a preregistered gate earns its keep. Without it, the attractive 9/10 headline could have justified promotion after the fact. With it, the question stayed boring: did the measured window meet the declared thresholds? No.

One metric cannot serve learning, operations, and promotion equally well

An online router wants a smooth signal. Fractional grades are useful because a 0.74 session should teach it more than a 0.26 session. Category correction can counter systematic grading bias. Cost correction can make a slightly weaker but much cheaper model competitive.

An operator wants event truth. Did the session finish? Did it produce a durable artifact? Did the process crash? Was the failure caused by the model, provider, or harness?

A promotion decision wants a stable acceptance contract. It may combine productive rate, raw quality, reliability, and cost, with thresholds fixed before seeing the sample.

Trying to compress all three into one number creates metric theater. If the posterior is treated as the operational ledger, transformations become invisible. If raw outcome alone drives the router, useful partial-quality information is thrown away. If the promotion gate is rewritten to match whichever signal looks best after the run, the pilot stops being an experiment.

Keep the layers separate:

session facts
  -> raw outcome + raw grade + failure attribution
  -> corrected online reward for routing
  -> preregistered multi-metric gate for promotion

Each transformation should be inspectable, and no aggregate should borrow the name of a stronger claim than it supports.

Names are part of the observability contract

total_rewards is technically defensible, but it invites the wrong inference. A dashboard should label it as something like reward-above-threshold count and show the threshold and reward source beside it. The same view should expose productive count, raw mean grade, corrected mean reward, and infrastructure failures separately.

That is not cosmetic wording. Operators make decisions from names before they read implementation details. A field that requires source-code archaeology to interpret is an observability bug waiting to happen.

The durable lesson from this pilot is not that bandits are misleading. It is that internal optimization signals must not impersonate ground truth.

The bandit said GLM-5.2 had often received enough corrected credit to strengthen its arm. The session ledger said only four of ten runs were productive. The promotion gate said do not promote.

All three statements can be true. Good systems make that obvious before someone acts on the shortest one.