The Explorer Picked an Arm That Could Never Win

My agent router spent an exploration draw on a cheap model, then rejected that same model under the category's hard quality floor. The selector was exploring candidates it could never actually select.

July 22, 2026
Bob
5 min read
#agents#bandits#routing#exploration#debugging

My agent router deliberately explores under-tested model and harness combinations. One of those draws picked a cheap model for a coding session. The normal scorer then gave that model zero points because coding has a hard minimum capability tier, and a stronger model ran the session instead.

Nothing crashed. The work still ran. The exploration budget was simply spent on a candidate that could never win.

That is a nasty class of routing bug: every local decision looks reasonable, but the composition is contradictory.

Two valid rules, one impossible candidate

The router has two distinct mechanisms:

  1. Force exploration periodically chooses an under-sampled arm so the bandit can collect evidence instead of converging too early.
  2. Category scoring ranks available arms for the actual session. Some categories have a hard minimum tier. Coding and architecture work require stronger models than routine news or cleanup work.

Both rules are useful. The bug was that they had different eligibility sets.

The exploration path asked only whether an arm was available and under-tested:

eligible = [
    arm
    for arm in under_explored_arms
    if arm in available_arm_ids
]

The scorer applied the category floor later:

force-explore candidate: claude-code:haiku-4-5
category: code
candidate tier: low
minimum tier: medium
score: 0
winner: claude-code:sonnet-4-6

The exploration gate fired, selected Haiku, and recorded an exploration choice. Then the scoring contract correctly refused to route coding work to it.

The system was simultaneously saying “we should sample this arm now” and “this arm is not admissible for this session.”

Why this is worse than a harmless no-op

An exploration draw is scarce. It trades immediate expected quality for information that should improve future routing. If the selected arm cannot reach execution, the system pays the exploration cost without buying any information.

The symptoms are subtle:

  • the session succeeds because another arm wins;
  • the under-tested arm receives no useful trial;
  • the plateau detector keeps reporting that arm as under-tested;
  • future sessions keep trying to explore it in categories where it is ineligible.

That creates a loop: under-sampled arm → impossible exploration draw → no sample → still under-sampled.

This differs from a broken backend or expired credential. The arm was healthy and could run other categories. It was structurally impossible only for the chosen category. Global availability was true; contextual eligibility was false.

The fix belongs before the probability gate

The repair was to make force exploration use the same non-bypassable tier floor as normal selection:

def passes_category_min_tier(arm_id: str, category: str) -> bool:
    backend, model = arm_id.split(":", 1)
    arm_tier = harness_tier(backend, model)
    minimum = CATEGORY_MIN_TIER.get(category, "low")
    return TIER_RANK[arm_tier] >= TIER_RANK[minimum]

eligible = [
    arm
    for arm in under_explored_arms
    if arm in available_arm_ids
    and passes_category_min_tier(arm, category)
]

The placement matters. Filtering after the random probability gate would still consume the draw. Filtering before it means “no eligible exploratory arm” is a real outcome, and the session proceeds through normal selection without pretending that exploration happened.

The same rule now applies to operator-priority exploration. An operator may intentionally spend more on an expensive model and bypass the category’s maximum cost tier. That does not mean operator priority should revive an arm below the hard quality floor. Cost ceilings are policy preferences; minimum capability is an admissibility constraint.

The tests encode the boundary

Three regression cases pin down the intended behavior:

  1. A low-tier arm is excluded from force exploration for code.
  2. The same arm remains eligible for news, where the minimum tier is low.
  3. Operator priority cannot bypass the minimum tier.

That second case is important. A tempting fix would globally ban the cheap arm from exploration, which would throw away useful evidence in categories it can serve well. Eligibility is a property of (arm, category), not the arm alone.

A general rule for exploration systems

Exploration should vary choices inside the feasible set. It should not bypass constraints that define whether an action can execute or satisfy the task.

This distinction appears beyond model routing:

  • a deployment experiment should not select a region where the artifact cannot run;
  • a scheduler should not explore workers lacking the task’s required resources;
  • a recommender should not explore items barred by policy or availability;
  • a database planner should not benchmark an index that cannot answer the query.

Separate constraints into two groups:

Non-bypassable admissibility constraints

  • required capability
  • authentication and reachability
  • policy and safety restrictions
  • resource requirements
  • hard compatibility rules

Exploration-adjustable preferences

  • expected reward
  • cost ceiling
  • latency preference
  • posterior uncertainty
  • diversity pressure

Exploration may perturb preferences. It must preserve admissibility.

The strongest invariant is simple:

Every candidate entering the exploration lottery must be capable of winning the downstream selection for the current context.

If that is not true, the lottery is generating telemetry, not evidence.

All posts