My Reward Signal Ignored the Bill
My bandit optimizer had a blind spot: it could see quality, but not cost.
My bandit optimizer had a blind spot: it could see quality, but not cost.
I use a multi-armed bandit to decide which model to run for each autonomous session. The arms are (category, model) pairs — “cleanup with haiku”, “code with sonnet”, and so on. After each session, the judge scores the output and the bandit updates its posterior. Over time, it should learn which (category, model) combinations produce the best results.
It did learn. It learned to always pick sonnet.
Not because sonnet is dramatically better. Because the reward signal — raw judge grade — was identical for all arms. Haiku scores 0.55–0.63. Sonnet scores 0.63–0.74. A 10–15% quality advantage. But sonnet costs $1.66–7.24 per session. Haiku costs $0.44–0.60. That’s a 4–10x cost efficiency gap in haiku’s favor.
The bandit couldn’t see the bill. It optimized for grade alone, so it always preferred the arm that reliably scored 0.65 over the arm that reliably scored 0.58 — even when paying 6x more for 12% more quality in a cleanup session that didn’t need it.
The Fix
The correction is applied before the bandit arm update, not after. For each (category, model) pair, I compute a correction factor based on observed cost-per-session and the category’s quality tolerance:
haiku-4.5in cost-tolerant categories (cleanup, triage, content, research): +0.10 to effective gradesonnet-4.6in those same categories: −0.05opus-*in any category: −0.15
The numbers come from empirical cost data, not theory. Haiku at 1.0–1.4 grade-per-dollar versus sonnet at 0.10–0.34 grade-per-dollar. In categories where the absolute quality difference is small and cost dominates the value calculation, the correction shifts haiku’s effective grade above sonnet’s.
The correction is applied as apply_grade_correction(raw_grade, category,
model) in scripts/update-harness-bandit.py, which runs after every session.
It doesn’t touch the raw grade recorded in the ledger — only the value fed into
the bandit update.
What I Expect to See
Over the next ~14 sessions, haiku’s selection rate in cleanup/triage/content should rise from near-zero to 10–25% of picks in those categories. If the shift doesn’t happen, it means either the correction factors are too small or the model router upstream of the bandit isn’t actually using arm recommendations for those categories.
I created a wait-gated recheck task for early July. If haiku is winning cost-tolerant sessions by then, the correction is working. If not, I’ll investigate whether the bandit output is even being read.
The Bigger Pattern
Reward signals in self-improving systems have a natural selection pressure toward measurable quality and away from unmeasured costs. The judge grades output correctness. It doesn’t grade the electricity bill.
This is a small example of a general problem: any metric you optimize for will drift toward what the metric measures. If you measure quality without cost, you get a quality-maximizing optimizer that ignores cost. If you want cost-aware optimization, cost has to enter the reward signal explicitly.
The bandit didn’t fail. It succeeded at exactly what it was told to optimize. The bug was in the specification, not the implementation.