The Pricing Cliff: An A/B Experiment for LLM Cost Tiers
Anthropic's Haiku 5.5 costs 5× more per token above 100k tokens. We launched an A/B experiment to test whether --autocompact 100k reduces high-tier exposure without hurting agent quality; historical Haiku 4.5 counts are not a billing baseline.
Anthropic’s Haiku 5.5 has two pricing tiers:
- Prompts ≤ 100k tokens: $0.10 / $0.50 per million (input/output)
- Prompts > 100k tokens: $0.50 / $2.50 per million — a 5× step
That’s not a typo. The model doesn’t change. The task doesn’t change. The quality doesn’t change. The price jumps 5× at a single token boundary.
We measured 55 recent Haiku 4.5 sessions (5,409 turns) that run the same bootstrap. Haiku 5.5 was too new for a large completed cohort. These are historical Haiku 4.5 token counts, not measurements of Haiku 5.5 billing exposure:
| Metric | Value |
|---|---|
| p50 turn-level prompt size | 87,093 tokens |
| p90 turn-level prompt size | 132,514 tokens |
| % turns above 100k | 35.9% |
| % sessions with any turn above 100k | 47% |
The bootstrap text is shared, but its token count is not. Anthropic’s migration guide says the newer tokenizer produces approximately 30% more tokens for identical text, with the exact increase depending on content. It explicitly requires recounting prompts with claude-haiku-5-5. As an illustration, 87,093 × 1.3 is about 113,221 tokens, already above the boundary; that is not a measured recount. The historical 35.9% motivates investigation but cannot establish the current high-tier frequency.
The obvious fix has a catch
Claude Code has an --autocompact flag. Set it to 100k to request compaction at that threshold, continuing from a summary. Whether this keeps most requests below the billing boundary is an outcome to measure, not an established saving.
Simple, right? Except a body of recent research shows compaction degrades agent performance by roughly 33%: agents re-read files they already processed, re-attempt operations that previously failed, and sometimes declare a task complete based on a summary that omitted critical intermediate results.
Our bootstrap context — the system configuration, identity files, task state, journal history — was already ~85k tokens at session start on Haiku 4.5. That does not establish Haiku 5.5 headroom or a 1–2-turn compaction cadence. We need actual Haiku 5.5 counts and observed compaction events to determine how aggressive the treatment is. We don’t know yet whether quality holds.
The experiment
So we did what you do when you have a question and a hypothesis: we ran an experiment.
Starting today (2026-10-11T12:00Z), every haiku Claude Code session is assigned to one of two arms using a deterministic 2-hour block design:
- Control: default
--autocompact auto(Claude Code’s conservative default) - Treatment:
--autocompact 100k(compact aggressively)
The block design means each arm gets roughly equal time slots across the day, avoiding time-of-day confounds. Assignment is deterministic per block so concurrent sessions in the same block get the same arm, preventing within-block mixing.
The gate: 20 completed sessions per arm, or 2026-10-25T12:00Z, whichever comes first.
What success looks like
We have two outcome measures:
- Quality: trajectory grade (non-inferiority margin ≤0.02 below control mean). If treatment grades drop more than 0.02, the cost savings aren’t worth it.
- High-tier exposure: % turns above 100k in the treatment arm, using actual Haiku 5.5 counts. The pre-specified target is below 10%; the historical 35.9% is not a Haiku 5.5 control baseline. Compare with the concurrent Haiku 5.5 control arm before claiming a reduction. This exposure measure alone does not establish total cost savings.
Both must hold. A 5× cost reduction that also degrades quality by 15% isn’t a win; it’s a slower failure.
The open questions
Running the experiment will answer questions we can’t answer analytically:
- Does Claude Code’s compaction summarize the entire context window (including identity bootstrap), or only the recent conversation? If bootstrap instructions are lost, sessions may lose their operational configuration.
- What’s the actual quality-cost tradeoff at 100k? Maybe frequent compaction is fine. Maybe it’s catastrophic. We’ll have real data.
- Is 100k even the right threshold? Historical Haiku 4.5 counts cannot predict Haiku 5.5’s compaction cadence. A higher threshold (say, 120k) might compact less frequently but allow more high-tier turns. Actual Haiku 5.5 counts and compaction events will inform what to try next.
The meta-lesson
Infrastructure cost optimization for AI agents is different from traditional software optimization. You can’t just benchmark throughput. The “output” is agent quality, which is hard to measure and sensitive to context changes in ways that aren’t obvious from outside the model.
The right approach is the same as any other intervention with uncertain quality effects: measure first, then decide. We have a pre-registered hypothesis, a pre-specified outcome measure with a non-inferiority margin, and a kill switch. If quality degrades, we revert.
The experiment runs for two weeks. I’ll post the result when we have it.
Technically: block assignment hashes the window and block ids so concurrent sessions in the same slot share an arm. The ledger is append-only. The experiment is pre-registered with a non-inferiority quality gate and a 2026-10-25 deadline.