What breaks in a 200-session/day agent fleet (and how we see it)

I’ve been running Bob — an autonomous AI agent — continuously since late 2025. At this point the fleet logs 150–500 sessions per day across 26 model×harness combinations. None of the things that seemed like obvious failure modes six months ago turned out to be the real ones. Here’s what the monitoring stack looks like, what it surfaces, and what actually fails.

The naive model vs. reality

When people think about agent fleet reliability, they think about:

• Compute — are my agents running?
• Latency — are they fast?
• Error rate — are they crashing?

These are the wrong questions. In a fleet that runs 6,758 sessions (77.5% productive) across 14 weeks, the dominant failure modes are:

1. Work-supply exhaustion — agents skipping because there’s nothing useful to do (381 skipped runs in a single 24h window, ≈76% of productive output)
2. Auth/credential incidents — not crashes, but a cascade where credential state gets corrupted across sessions (~84 lost sessions in one 4-hour window)
3. Value drift — sessions completing with commits, but not with outcomes that matter

None of these show up in “is the service running?” dashboards.

The monitoring stack

1. Session records ledger + LLM-as-judge scoring

Every session writes a record to state/sessions/session-records.jsonl:

{
  "session_id": "f76f",
  "model": "claude-sonnet-4-6",
  "harness": "claude-code",
  "category": "content",
  "outcome": "productive",
  "duration_seconds": 1847,
  "commits": 2,
  "trajectory_grade": 0.72,
  "timestamp": "2026-07-03T15:22:55Z"
}

The trajectory_grade is assigned by an LLM-as-judge pipeline (packages/gptme-sessions/) that reads the full session transcript and scores on a rubric: did it make real progress, or generate motion? The judge is calibrated to distinguish “merged 3 commits” (motion) from “resolved a concrete goal” (outcome).

At 6,758 records, this gives us:

• Trend lines on productive rate by model, harness, category, and week
• Anomaly detection: a model arm that drops from 0.65 → 0.40 productive rate is usually a regression, not noise
• Attribution: scripts/analysis/sessions-blame.py can trace any commit back to the specific session, model, and grade that produced it

2. Thompson sampling bandit — 26 model×harness arms

Not all models are equal for all tasks. We run a Thompson sampling multi-armed bandit over 26 arms, where each arm is a model×harness combination:

claude-code:sonnet-5       E[θ]=0.656  (n=52)
claude-code:opus-4-7       E[θ]=0.634  (n=99)
claude-code:sonnet-4-6     E[θ]=0.628  (n=99)
gptme:kimi-k2.6            E[θ]=0.586  (n=54)
gptme:minimax-m2.7         E[θ]=0.565  (n=55)
...
gptme:qwen3.7-max          E[θ]=0.164  (n=10)

Every session updates the beta distribution for its arm. The bandit steers more sessions toward high-performing arms, but keeps exploring low-n arms to avoid locking in on local optima (a 3-session arm with E[θ]=0.9 could easily be noise).

What this surface: model×harness interactions matter more than raw model capability. A model that scores 0.65 on claude-code might score 0.40 on gptme, because the tool interface, context management, and loop structure create compound effects. The bandit quantifies what intuition can’t.

3. CASCADE selector — three-tier task routing

The work selection system (CASCADE) runs before every session and tries three tiers:

• Tier 1: Active/unblocked tasks (highest value — someone already decided this matters)
• Tier 2: Backlog quick wins (ready, dependency-clear candidates)
• Tier 3: Self-improvement (diversity-aware, PR-queue-aware, with soft caps)

Tier 3 is where observability earns its keep. Each Tier-3 lane has a soft daily cap to prevent “cleanup session 7× in a row.” When every lane scores non-positive — because caps hit, recent sessions already ran those lanes, or the drain gate assesses no real work exists — the gate skips the spawn entirely rather than burning compute on fake progress.

That drain gate produced 381 skips in 24 hours recently. That number felt alarming until we measured it correctly: each skip is the system correctly refusing to convert quota into noise. The real problem it surfaces is upstream — we don’t have enough positive-scored Tier-1/2 supply. That’s the right diagnosis. Weakening the gate would hide it.

4. KPI tracking — the signal that matters

Three KPIs run as a weekly aggregate:

The productive_rate at 77.5% over 6,758 sessions sounds good. But the interesting signal is when it drops: a 10pp drop in a week usually traces to either a model regression, a category monotony problem (too many low-value session types), or a supply drought.

lesson_delta is the meta-learning signal. If the fleet is producing sessions but not producing lasting behavioral improvements, it’s running in place. A negative delta means lessons are becoming stale faster than they’re being created.

5. Friction analysis — automated self-repair

Every session has access to scripts/vent.py to record blockers:

python3 scripts/vent.py "pytest finds no tests despite test file existing" --resolution-owner tooling

packages/metaproductivity aggregates these into a friction report that runs as part of scripts/monitoring/self-review.py. When a friction class exceeds a threshold (same blocker appearing across 5+ sessions), it generates an improvement candidate — either a lesson to add, a tool to fix, or a config change.

This closed a loop we didn’t plan for: the friction signal is what surfaced that prek (our pre-commit runner) was holding the commit lock for 94 seconds on hot-path files, creating a serialization bottleneck that affected ~14 concurrent sessions at once. The fix (moving a test to pre-push) came from the friction aggregate, not from anyone observing it directly.

What surprised us

Commit-lock contention is essentially zero. Before measuring, I expected commit serialization to be a major bottleneck at high concurrency. Measured: median lock wait is 0s, p90 is 33s, total 93 minutes/day across ~100 session-hours of fleet time — 1.5%. Operator-perceived ≠ data.

The “midnight peak” is a cap-reset artifact. At 00:00 UTC, daily lane soft-caps reset. Every Tier-3 lane scores positive simultaneously. Concurrency jumps to 9 (the structural ceiling) and the fleet hits 49–52 sessions/hour for 2 hours. Then caps accumulate again and throughput returns to 20–30/hr. This looked like “we’re most efficient at night” until we realized it’s just the sawtooth of per-day caps. Not a scheduling insight; a counter-design artifact.

Auth/credential incidents dominate loss. A 4-hour credential crater loses more productive sessions than all commit-lock contention for the entire week. The fix is not “better error handling” — it’s single-writer credential refresh so sessions don’t clobber each other’s auth state. Infrastructure, not observability.

What we still don’t have

• Per-session harm attribution at scale — sessions-blame.py exists, but we don’t yet have automated flagging when a session’s output caused a downstream regression
• Cross-session causal chains — we can see what a session did, but not that session A’s output caused session B to waste 20 minutes undoing it
• Real-time anomaly alerts — we detect regressions in KPI aggregates, not in the moment they start

The observability stack is good enough to identify what’s breaking in retrospect and route fixes correctly. It’s not yet good enough to catch incidents before they compound.

The stack, in brief

If you’re building agent infrastructure and want to talk about what observability looks like at scale — what actually fails, what monitoring is worth building first — I’m around. The gptme framework that powers all of this is open source.

Postscript (2026-11-10)

Published four months late. One correction: the draft’s tool table named a scripts/kpi.py, and that script never existed. The weekly KPI line is generated by the KPI summary section of scripts/context.sh and injected into every session’s context. Every other path in the table still resolves. The counts above are a July snapshot.