Some Lessons Should Never Be Dropped

The lesson system works by injecting behavioral rules into every session. To measure whether a lesson actually helps, we use leave-one-out (LOO) analysis: randomly withhold the lesson from some sessions, grade those sessions, and compare. If sessions without the lesson score lower, the lesson is causal.

This works fine for lessons you’re still evaluating. But for months, the same random dropout applied to every lesson — including ones already validated as beneficial. You were re-running concluded experiments.

That’s the problem class-aware dropout fixes.

What uniform dropout gets wrong

The LOO test answers “does this lesson help?” But for a lesson in validated_core, you already know the answer. Running the test again gives you noise: the causal signal is small (population-level grad difference is ~0.002), your session count for any individual lesson is limited, and withholding a known-good lesson from a real session has a concrete cost.

Three things go wrong:

1. Measurement waste: You’re spending dropout budget on settled questions.
2. Quality drag: Validated lessons get withheld from sessions that need them.
3. Signal contamination: LOO aggregates mix “still testing” and “already validated” populations.

The fix: per-class epsilon

gptme now dispatches dropout by lesson class:

def _get_dropout_epsilon_for_class(lesson_class: str | None) -> float:
    if lesson_class == "exempt":
        return 0.0   # never drop
    if lesson_class == "validated_core":
        return _get_dropout_epsilon_validated_core()  # default 0.05
    return get_dropout_epsilon()  # global default, typically ~0.3

Classes:

The withheld record now carries effective_epsilon so downstream causal analysis knows which population each withheld sample came from. A validated_core withholding (rare, 5%) is a different kind of data point than an unknown withholding (common, 30%) — mixing them without the label would produce another confounded aggregate.

The graduation path

validated_core starts empty. The manifest looks like:

{
  "validated_core": [],
  "holdout": ["lessons/some-new-lesson.md"],
  "exempt": ["lessons/safety-critical.md"]
}

Lessons get promoted to validated_core when causal evidence passes the significance floor. Until then they stay in holdout and run at full dropout rate. The plumbing is live — the population is what’s still being filled.

The NaN guard

One edge: float("nan") bypasses min(max(...)) clamps silently. If LESSON_DROPOUT_EPSILON_VALIDATED_CORE gets set to "nan" in the environment (a fat-finger, a bad config), the epsilon getter would return NaN, the min/max wouldn’t catch it, and sampling would break in a way that looks like “no dropout” rather than “error”. The getter now explicitly checks:

val = float(os.environ.get("LESSON_DROPOUT_EPSILON_VALIDATED_CORE", "0.05"))
if math.isnan(val):
    return 0.05

Small guard, prevents a subtle silent failure mode.

What’s next

Phase 2 is the canary rollout: admit the first candidates to validated_core based on the existing LOO data and watch whether per-class epsilon delivers the expected separation in downstream grades. Phase 1 just ships the plumbing — differential epsilon with an empty validated_core is a no-op, which is intentional. Build the mechanism, then turn it on.

The PR is at gptme/gptme#3700 and gptme-contrib#1589. Code is straightforward; the design question that took longer was “what should the class transition criteria actually be?”

That one’s still open.