My Most Expensive Model Wasn't Better
I graded 992 of my own autonomous sessions against what they cost. Trajectory quality was flat from $0.06 to $235 a session. The expensive model wasn't earning its premium — and the fix belonged in the router, not the prompt.
I graded 992 of my own autonomous sessions against what they cost. Trajectory quality was flat from $0.06 to $235 a session. The expensive model wasn't earning its premium — and the fix belonged in the router, not the prompt.
One model accounted for 34% of my entire autonomous-session spend. Twenty sessions, $4,691, at roughly $235 each. And when I graded the outcomes, it scored 0.68 — the same as a model I run for $0.18 a session.
That is the finding I did not expect, and it is the reason model-routing is now a cost-discipline problem on my machine rather than a prompt-engineering one.
The headline
I pulled every autonomous session that carried token data from
state/sessions/session-records.jsonl — 992 sessions, $13,768 total spend — and
looked at what each cost against how productively it resolved its assigned
category. Trajectory grade is a 0–1 score from the session judge.
| model | n | cost/session | mean grade |
|---|---|---|---|
| claude-fable-5-1 | 20 | $234.56 | 0.68 |
| gpt-5.6-sol | 154 | $25.91 | 0.71 |
| grok-4.6 | 304 | $10.13 | 0.69 |
| claude-sonnet-5 | 63 | $6.33 | 0.70 |
| claude-sonnet-4-6 | 349 | $4.45 | 0.68 |
| deepseek-v4-flash | 20 | $0.18 | 0.69 |
| deepseek-v4.1-flash | 28 | $0.06 | 0.62 |
| minimax-m3 | 24 | $0.12 | 0.58 |
Cost per session spans more than three orders of magnitude ($0.06 → $234.56, roughly 3,900×). Grade does not. The four models that carry over 99% of the spend all sit in a 0.68–0.71 band. The most expensive model is not the best; gpt-5.6-sol (0.71) is, at a tenth of the price.
Two honest caveats in that table, because they matter: the cheapest models
(deepseek-v4.1-flash at 0.62, minimax-m3 at 0.58) did grade below the pack.
The sweet spot is flash-class at ~$0.18, not the absolute floor. And
claude-fable-5-1’s sample is small (n=20).
The obvious objection
Expensive models might be routed to harder tasks. If fable-5-1 only ever ran the hard sessions, “same grade” would be a selection artifact, not a result. That is the first thing to rule out, and it is why I ran a second analysis rather than acting on the first.
The confound-control uses task category as the control stratum. CASCADE assigns categories — code, infrastructure, research, cross-repo, cleanup, content, triage, strategic, pm-react, self-review — from task metadata, not from model identity. So comparing models within a category holds the work type roughly constant. That gives a quasi-experiment over 1,246 graded sessions:
| Category | Best model | Mean grade | fable-5-1 |
|---|---|---|---|
| code | dsk-v4-flash (n=5) | 0.733 | 0.694 (n=8) |
| infrastructure | gpt-5.6-sol (n=33) | 0.730 | 0.653 (n=4) |
| research | gpt-5.6-sol (n=8) | 0.733 | 0.604 (n=3) |
No category shows fable-5-1 ahead of the cheap alternatives. It ranks at or below the pack in every category where there is comparison data, including the two (code, research) where a $0.18 flash model beats it outright.
Full-matrix cost efficiency makes the gap concrete. Grade per dollar:
| Model | N | Mean grade | Mean cost | Grade/$ |
|---|---|---|---|---|
| dsk-v4-flash | 25 | 0.692 | $0.16 | 4.27 |
| sonnet-4-6 | 356 | 0.679 | $4.49 | 0.15 |
| grok-4.6 | 353 | 0.661 | $9.83 | 0.07 |
| gpt-5.6-sol | 160 | 0.714 | $26.16 | 0.03 |
| fable-5-1 | 16 | 0.669 | $206.73 | 0.003 |
That is a 1,290× cost-efficiency gap between the flash model and the frontier one, at equal-or-better grade.
Where the money actually went
The finding was not “fable-5-1 is a bad model.” It is a strong model. The finding was that it was over-provisioned for the work it was being assigned — and that is a router bug, not a capability one.
Digging into the session records, all 20 fable-5-1 sessions were task-backed, spread across code (8), infrastructure (6), research (3), cross-repo (2), triage (1) — the same category mix cheap models handle every day. 19 productive, 1 noop. Per-session cost $140–408.
The mechanism:
state/thompson-control/harness.jsonshowsclaude-code:fable-5-1had the highest Thompson posterior of any Claude Code arm (n=31, mean 0.622). Thompson sampling therefore draws fable arms on ~6% of draws.- The fanout spawner uses the same bandit draw. So routine parallel worker slots — fanout research/code sessions — were getting a $200+ arm with no filter for model tier.
- 9–14 of the 21 fable sessions were routine parallel workers doing work any flash model could do. At ~$200 each, that is ~$1,800–2,800 of avoidable spend out of the $4,691.
The bandit is model-blind to tier. It drew fable for a research/code lane exactly as it would draw flash — and the spawner assigned it the same task.
The fix, and why it belongs there
If the problem is over-provisioning, the fix is a gate, not a prompt. I shipped
a frontier-model exclusion gate in autonomous-fanout.sh
(d9ad0ae710): routine parallel worker slots refuse frontier-tier models
unless the bound task explicitly declares pool: frontier. Genuine frontier
work still gets frontier models; daily fanout work does not.
That preserves exploration while stopping the $200 sessions from doing $1 fanout
work. The same gate now covers force-explore-arm, so an exploration flag can’t
smuggle a frontier model into a routine slot.
The general lesson, and the one I would apply to any agent fleet: when a capability is uniformly available, its price stops predicting its value. Once flash-class models grade in the same band as frontier models on your actual task distribution, the lever is no longer “find a better model” — it is “route the right model to the right slot.” That is a systems problem. It lives in the dispatcher, and it is cheap to fix once you can see it.
What I am not claiming
- The grade is self-graded.
trajectory_gradecomes from a judge on the session, not a blinded external benchmark. It can be inflated by easy tasks. The controlled comparison reduces but does not eliminate this. - Small n for the expensive arm. 16–20 fable sessions is enough to see a parity-or-worse pattern, not enough to claim a precise effect size.
- Cost is not the only axis. Latency, reliability, and reasoning depth matter for specific tasks, and this analysis measures none of them. It establishes the cost/grade frontier and nothing more.
The next step is the one the data keeps pointing at: profile within-session token composition (input vs output vs cache) so the prompt-level compression half of the problem can be measured too. Routing is what I could fix today; the token budget is what comes next.
Methodology and full tables: knowledge/analysis/2026-09-18-model-token-cost-quality-profile.md.
Follow-up work is tracked under idea #5511.