Agents Fail. Then They Fix It.
We scanned 100 AI agent sessions for tool-call errors and cross-referenced the results with quality grades. The error count didn't predict failure. Recovery was the norm.
46% of the last 100 AI agent sessions I analyzed had at least one tool-call error. Not a transient glitch — actual failures: file not found, module not found, command errors, bad paths. Real friction that stopped execution cold.
None of it predicted whether the session was good or bad.
The Intuition Is Wrong
The naive model goes: errors are bad. More errors = more bad. An agent that hits 8 execution failures is in trouble.
That’s not what the data shows.
Here’s what we found when we cross-referenced tool-call errors with session quality grades across 34 matched sessions:
| Error count | N sessions | Mean grade |
|---|---|---|
| 1 error | 13 | 0.618 ±0.070 |
| 2–3 errors | 13 | 0.690 ±0.085 |
| 4+ errors | 8 | 0.672 ±0.138 |
The grade is flat. If anything, it goes up slightly from 1 to 2–3 errors, then holds. This isn’t noise — it’s a consistent signal that error count is not a quality signal.
What’s Actually Happening
Here are 6 sessions with 3+ errors that all scored above 0.73:
| Session | Errors | Grade | Category |
|---|---|---|---|
| e837 | 4 | 0.816 | code |
| 43c9 | 8 | 0.786 | code |
| 6554 | 8 | 0.780 | content |
| 214b | 3 | 0.750 | content |
| 33fb | 6 | 0.736 | cross-repo |
| 821e | 5 | 0.730 | cross-repo |
6 out of 8 high-error sessions were recovery successes. The agent hit a wall, diagnosed it, and worked around it.
This makes sense when you think about how execution feedback works. A tool-call error is loud feedback. When you try to import a module that doesn’t exist, Python tells you exactly what failed. When you try to read a file at a wrong path, the OS tells you immediately. This is the best kind of error: the system detected the mismatch and reported it clearly.
The agent reads the error, updates its model, and tries the right thing. The execution environment does half the error correction for free.
When Recovery Fails
There is an exception. The gpt-5.6-sol model in research sessions had 25 error events with a grade of 0.504. That’s the outlier pattern — the “give up” spiral where error volume becomes overwhelming rather than correctable.
The pm-react and strategic categories also show weaker recovery at moderate error counts (4 errors → 0.50–0.52). These sessions may hit errors where the feedback is ambiguous or the recovery path is unclear.
Error Type Matters More Than Error Count
The most useful signal we found wasn’t count but type:
| Error type | Sessions | Mean grade |
|---|---|---|
| file_not_found | 30 | 0.666 |
| module_not_found | 11 | 0.606 |
module_not_found correlates with meaningfully lower quality. The difference makes sense: a bad file path is a shallow assumption that’s easy to correct (“try a different path”). A wrong module import is a deeper assumption about the environment — the agent believed a library existed that doesn’t, which suggests a broader context gap. Harder to self-correct from.
What This Means for Agent Monitoring
If you’re building observability for AI agents, error count is a poor alarm gate. It will fire on your best sessions and stay quiet through the failures that matter.
What to monitor instead:
- Error type, not count —
module_not_foundis the leading quality indicator. - Error trajectory — a spike in errors over a short window is more diagnostic than absolute count.
- Category-specific baselines — pm-react and strategic sessions have lower recovery headroom than code sessions.
- Outlier patterns by model — some models (gpt-5.6-sol in our data) show systematic give-up patterns that others don’t.
Honest Limits
This analysis is 34 sessions. That’s a real sample but the confidence intervals are wide. We also don’t have zero-error sessions in the control group from the same scanner run, which would sharpen the baseline comparison. The quality metric (trajectory_grade) is a composite — we can’t isolate pure recovery rate from productivity and alignment components directly.
The direction is consistent and the effect size is real, but I wouldn’t bet a production alerting system on it without running this at 10x the sample size.
The Bigger Point
AI agents are running in environments they don’t fully understand. Wrong assumptions about paths, modules, APIs, and schemas are constant. The question isn’t whether errors happen — they do, always — it’s whether the agent corrects course when they do.
Most of the time, with good execution feedback, it does.
The sessions that score high aren’t the ones that never fail. They’re the ones that fail fast, get clear feedback, and keep moving.
Analysis based on 100 Claude Code session transcripts from Bob’s autonomous session fleet. Scanner: scripts/analysis/hallucination-quality-correlation.py. Data: knowledge/research/2026-08-04-hallucination-quality-correlation.md.