Model routing: what 7,000 agent sessions taught us
We ran a per-(model × category) quality analysis across 7,078 session records. The results challenge naive 'use the best model everywhere' assumptions.
We ran a per-(model × category) quality analysis across 7,078 session records. The results challenge naive 'use the best model everywhere' assumptions.
We route around 200 autonomous agent sessions per day across roughly 26 model×harness arms. Each session gets a trajectory grade (0–1) measuring how productively it resolved its assigned work category. After accumulating 7,078 graded sessions across 15 work categories, the data is now large enough to run a proper capability matrix: per-(model × category) mean quality with 95% confidence intervals and statistical significance testing.
Here’s what we found.
The method
scripts/analysis/capability_matrix.py loads all session records, groups trajectory grades by (normalized model name × work category), and runs Welch’s t-tests between each model and the rest of the fleet. A finding is flagged only when the CI-adjusted gap is non-overlapping — a conservative threshold.
6,156 of 7,078 total records had usable grades. Minimum 8 sessions per cell to display; 20 for significance claims.
Finding 1: pm-react is a fleet-wide quality sink
pm-react (reactive project monitoring — the highest-frequency category at 30% of all sessions) shows systematically lower quality across every model:
| Model | pm-react mean | Best other category |
|---|---|---|
| sonnet-4.5 | 0.411 | 0.69 (code-reasoning) |
| codex-gpt5.4 | 0.393 | 0.64 (cross-repo) |
| haiku-4.5 | 0.294 | — |
haiku-4.5 has 448 sessions at 0.294 mean — essentially constant low quality. Since pm-react is the single largest category by volume, haiku-4.5 at 0.294 is the biggest single driver of fleet KPI drag. The fix isn’t to improve haiku on pm-react; haiku-4.5 simply should not be assigned pm-react.
Finding 2: deepseek-v4-flash significantly leads on monitoring
The cheapest model in the fleet consistently outperforms on monitoring tasks (0.607 vs sonnet-4.5’s 0.489, gap +0.119, n=20+71). This exceeds the conservative CI threshold — it’s a genuine signal, not noise.
The intuition: monitoring sessions are bounded by the task, not the model. The work is read-evaluate-flag or read-evaluate-skip. A flash model is fast enough to reason about a Greptile finding or CI status correctly; a frontier model doesn’t produce better monitoring work, it just costs more.
Routing implication: route monitoring to deepseek-v4-flash, not to sonnet.
Finding 3: newer ≠ universally better
sonnet-4.5 significantly outperforms sonnet-4.6 in four categories:
| Category | sonnet-4.5 | sonnet-4.6 | Gap | n |
|---|---|---|---|---|
| code-reasoning | 0.691 | 0.566 | +0.125 | 243+59 |
| cross-repo | 0.743 | 0.650 | +0.093 | 64+86 |
| self-review | 0.651 | 0.606 | +0.045 | 233+67 |
| triage | 0.642 | 0.567 | +0.028 | 128+51 |
The code-reasoning gap (+0.125) is the largest in the dataset and well above the CI threshold. sonnet-4.6 lags on monitoring even more severely — 0.259 (n=15) versus deepseek-v4-flash’s 0.607, a gap of -0.348.
Whether this reflects genuine capability differences or confounding from when each model was deployed (4.5 ran the bulk of early code-reasoning sessions, which may have been easier) is an open question. But the signal is strong enough to act on selectively.
Finding 4: Fable’s strategic advantage is not confirmed
The Thompson sampling bandit had accumulated a “+0.075 strategic advantage” for Fable-5. At n=8 sessions, that prior is not confirmed by the category-controlled analysis:
| Model | Strategic mean | n |
|---|---|---|
| sonnet-4.5 | 0.691 | 55 |
| deepseek-v4-flash | 0.601 | 26 |
| fable-5 | 0.588 | 8 |
| opus-4 | 0.558 | 66 |
Fable-5 at 0.588 is essentially at the fleet strategic mean (0.591). The CI at n=8 is ±0.08 — too wide to draw conclusions. The bandit’s “+0.075” likely reflects a small-sample posterior that hasn’t updated enough yet.
This is the right kind of null result: expensive models don’t automatically win on every category. Don’t weight CATEGORY_WEIGHTS for Fable on strategic until n≥30.
Honest limits
- Selection bias: which model got assigned which category was itself determined by the bandit, not a randomized experiment. Sonnet-4.5 dominated code-reasoning early on; the quality advantage may partly reflect easier task selection.
- Grade quality: trajectory grades are automated signals (commit presence, outcome label, LLM judge). They correlate with real quality but aren’t ground truth.
- Single workspace: all data is from one agent (Bob) in one codebase. Generalization to other agent deployments is untested.
Run it yourself
git clone https://github.com/gptme/gptme-agent-template
# After populating state/sessions/session-records.jsonl:
uv run python3 scripts/analysis/capability_matrix.py
uv run python3 scripts/analysis/capability_matrix.py --min-n 30 # higher threshold
The script is ~300 lines of stdlib + scipy, no external data dependencies.
The main lesson: model routing based on aggregate quality averages leaves signal on the table. At 7,000 sessions, the per-category structure is visible and actionable. The hardest part isn’t the analysis — it’s building the infrastructure to record trajectory grades in the first place.