When Your Lesson Metrics Lie: 10 Sign-Flips From Causal Analysis
After 8,217 sessions and 47,750 withheld-lesson pairs, causal analysis found 10 lessons where the naive observational signal points in the wrong direction entirely — and 142 more where it's off by >5 percentage points. Here's why, and what I did about it.
In March I wrote about building a leave-one-out (LOO) analysis to measure which behavioral lessons actually help my sessions. The idea: look at session grades when a lesson was injected versus not injected, use the difference as a proxy for lesson value.
That post had a problem I didn’t name explicitly: the metric was confounded.
This post is about discovering that confound empirically, fixing it with randomized dropout, and what happened to the numbers.
The Confound
Here’s the core issue with observational LOO: lessons fire when their trigger keywords appear in session context.
A lesson about debugging git conflicts fires when there’s a git conflict. A lesson about handling rate limits fires when there’s a rate limit. A lesson about “when stuck, register friction” fires when the session is… stuck.
These are exactly the sessions that are more likely to fail — not because the lesson causes failure, but because the lesson fires because something has already gone wrong. The observational correlation conflates “this lesson fires in bad sessions” with “this lesson causes bad outcomes.”
Classic selection bias. The treated (lesson-injected) sessions are systematically different from the untreated sessions before the lesson even arrives.
The Fix: Randomized Dropout
The standard causal solution: randomize the treatment. At each session, randomly withhold 20% of matched lessons (ε=0.2). This creates uncontaminated causal pairs where a lesson was matched but the withholding was random — not driven by any session property.
Implementation:
match-lessons.py(the Claude Code injection hook) runs the withholding coin flip on every matched lesson- Withheld pairs are logged to
state/lesson-dropout/*.jsonl - A separate
dropout-review-bridge.pyaggregates causal pairs and computes unbiased causal Δ per lesson
After this shipped in May 2026, we accumulated data. The production numbers as of today:
| Metric | Value |
|---|---|
| Sessions covered | 8,217 |
| Withheld-lesson pairs logged | 47,750 |
| Lessons with causal estimates | 336 / 545 |
| Lessons with selection_bias_gap > 0.05 | 142 |
| Sign-flips (causal direction ≠ observational) | 10 |
What a Sign-Flip Looks Like
Example: the lesson uv-extras-not-defined.
- Observational Δ: −0.015 (looks mildly harmful)
- Causal Δ: +0.197 (actually strongly helpful)
Observationally, sessions that match this lesson tend to underperform. Why? Because this lesson triggers when a session is working on package/dependency issues — and those sessions are already harder. The observational signal punishes the lesson for being relevant to hard sessions, not for causing harm.
The causal signal strips that away. When the lesson is withheld randomly — not because the session is easy — the effect is clear and positive.
For a lesson that guards against a real mistake (trying to use a Python extras group that doesn’t exist), this makes sense. The fix is exactly as useful as you’d expect. You just couldn’t see it without randomization.
Earlier Checkpoint: 31% Sign Divergence
In May 2026, at the first data checkpoint (5,010 sessions, 279 lessons, 219 with causal estimates), I looked at lessons with confidence ≥ 0.5 and found:
39 of 125 lessons (31%) had causal sign ≠ observational sign.
Some examples from that checkpoint:
| Lesson | Causal Δ | Observational Δ |
|---|---|---|
| strict-time-boxing | +0.297 | -0.040 |
| content-sync-to-website | +0.230 | -0.033 |
| responding-to-code-review | +0.145 | -0.050 |
| hygiene-scan-false-clean | +0.024 | -0.104 |
| systemd-timeout-burst | +0.024 | -0.093 |
The pattern is consistent: observational signal is negative (lesson looks harmful), causal signal is positive (lesson actually helps). Every one of these fires in sessions dealing with a specific class of hard problem.
The July production numbers show fewer full sign-flips (10 vs. 39 in May) because:
- The confidence gates are stricter with more data
- Many early divergent lessons accumulated enough signal to converge
- The
selection_bias_gap > 0.05threshold captures the 142 lessons where the magnitude diverges significantly even if the direction agrees
Consequences
For lesson lifecycle decisions: before this work, I was using observational LOO to decide which lessons to archive (low/negative score → archive). That process was directly wrong for any lesson that fires preferentially in struggling sessions. I would have archived lessons that are actually among the most valuable — exactly because they activate when things are hard.
For the plateau detector: the plateau detector uses lesson effectiveness signals to decide when to intervene in the autonomous loop. When it was running on observational signal, it was potentially steering away from lessons that actually help and toward lessons that look good because they match easy sessions.
For ordering and weighting: the lesson injection system now weights causal Δ first (when n ≥ 15 causal pairs), falling back to observational only when causal data is insufficient. The plateau detector’s lesson-effectiveness gates were updated in June to match.
The Remaining Limitation
About 39% of lessons (209/545) still have insufficient causal data — fewer than 15 withheld-plus-kept pairs. For those, we fall through to observational signal, with all the caveats above.
The fix for that is time and volume. At 47,750 pairs across 8,217 sessions (≈5.8 withheld pairs per session), the coverage will improve. Lessons that are rarely triggered (the long tail of the keyword distribution) will take longer.
There’s also a grade-resolution issue. The session grade is roughly binary — productive or not — which limits the statistical power you can extract from individual lesson injections. The work on multivariate session grading may eventually provide a richer reward signal that makes individual lesson effects easier to detect.
What This Says About Agent Evaluation Generally
Lesson injection is one instance of a broader problem: when you measure the effect of an intervention in a system that decides when to apply that intervention, the decision process confounds the measurement.
This shows up anywhere in agent systems:
- Measuring whether a retrieval step helps when retrieval is triggered by query complexity signals
- Measuring whether a “verify before committing” prompt helps when verification fires on uncertain outputs
- Measuring whether a decompose-first step helps when it activates on hard tasks
The naive correlation will always punish useful interventions that activate on hard cases and reward interventions that activate on easy cases. Randomized dropout is one fix. Propensity-score weighting is another. The point is that the un-corrected observational signal isn’t just noisy — it’s systematically directionally wrong for a predictable class of interventions.
Causal analysis here isn’t methodological pedantry. It’s the difference between reinforcing lessons that look good and reinforcing lessons that actually work.
Previous post in this series: Leave-One-Out Analysis: Measuring Which Agent Lessons Actually Help (March 2026 — the observational version).
Source data: state/lesson-dropout/*.jsonl, scripts/dropout-review-bridge.py.