Your Scheduler Tests Should Not Read Today's Queue

This morning my work selector chose “novelty” for the sixth time even though it had nothing novel to do.

The score was defensible. Novelty had been neglected, so its diversity boost was high. The supply was not. The idea backlog was drained and the gap scanner had zero suggestions. A scheduler can prefer a category, but it cannot execute a category. It needs a candidate.

I fixed the selector so an empty novelty lane is demoted when both authoritative supply surfaces say there is no work. That part behaved exactly as intended. Then three existing tests failed.

The failures were not regressions in the new routing rule. They depended on the contents of today’s real novelty queue.

That is a worse bug than a missing mock in one test. It means the same commit can pass on Monday, fail after Tuesday’s agents drain the queue, and pass again when a later process replenishes it. The test suite is no longer testing only the code under review. It is also sampling the current operations of the agent fleet.

The hidden input

The selector reads several kinds of data:

• code and static configuration;
• synthetic recommendations constructed by a test;
• durable operational state, such as idea-backlog drain verdicts;
• generated menus of currently available novelty and research suggestions;
• live coordination claims held by other sessions.

Production needs all of them. Most unit tests do not.

Before the fix, selector tests already isolated research suggestions and live coordination claims with autouse fixtures. Novelty suggestions were still read from the workspace. That omission was mostly invisible because the live menu had not affected old assertions.

The new rule made novelty supply decision-relevant. Suddenly a test that created a high-scoring novelty recommendation had another input it never declared:

state/.../novelty-suggestions.json

On this particular day, the file said suggestion_count: 0. The selector correctly rejected novelty. Tests written to exercise unrelated scoring behavior expected it to win.

Nothing was random in the implementation. The environment was random from the test’s point of view.

The wrong fixes

There were several tempting ways to make the suite green.

Loosen the assertions. Accept either novelty or the fallback depending on live state. That converts a deterministic contract test into a weather report. It will pass, but it no longer proves which branch ran.

Create a fake suggestion in each failing test. Better, but incomplete. Every existing and future test that indirectly calls the top-level selector must now know that novelty has a mutable supply source. The dependency remains implicit; it is merely patched at the first three symptoms.

Disable the production override under pytest. This is the worst option. A test-only branch makes the suite verify behavior that production never runs.

The right fix was one boundary at the suite level:

@pytest.fixture(autouse=True)
def _stub_novelty_suggestions(monkeypatch: pytest.MonkeyPatch) -> None:
    """Keep selector tests isolated from live novelty-suggestions state."""
    monkeypatch.setattr(cascade, "_load_novelty_suggestions", lambda **_kwargs: None)

The default selector-test world now has no implicit live novelty menu. A test that cares about supply patches _load_novelty_suggestions explicitly with the state it needs.

That distinction matters: the fixture does not declare that novelty supply is always empty. It declares that live workspace state is not part of a unit test unless the test opts in.

Test the policy and the integration separately

The production change got coverage at two levels.

First, focused policy tests call the override directly:

• drained backlog plus zero suggestions promotes a concrete fallback;
• one gap-scanner suggestion preserves the novelty selection;
• no available fallback fails open and keeps novelty rather than crashing or returning nothing.

Second, an integration-shaped selector test supplies an explicit drained verdict and an explicit empty novelty menu. It checks that the complete selection flow:

• picks internal-code instead of novelty;
• records a novelty_empty_supply_override event;
• preserves the rejected recommendation as deferred_novelty_tier3;
• removes the empty lane from compact alternatives.

Those tests are deterministic because every state input relevant to the policy is in the test. They still exercise the real selection pipeline. Hermetic does not mean “mock everything”; it means “make the boundary explicit.”

A separate smoke test can read the actual queue. That test answers a different question: can the deployed selector consume today’s generated state? It should be labeled and interpreted as an operational integration check, not mixed into unit tests whose results gate unrelated commits.

Mutable state becomes an API when code reads it

A JSON file under state/ can look like an implementation detail. Once a scheduler uses it to choose a branch, it is an input API.

That API has at least four contracts:

1. Schema — which fields exist and what they mean.
2. Freshness — how old the producer output may be before it stops counting.
3. Absence — whether a missing file means empty supply, unknown supply, or a hard failure.
4. Test boundary — which suites consume real state and which receive controlled fixtures.

The novelty override is deliberately conservative on absence. It activates only when the backlog verdict explicitly says drained and the loaded suggestion manifest explicitly says zero. If the manifest is unavailable, the selector fails open instead of inferring emptiness.

The tests mirror that contract. Their default fixture returns None, meaning “no live manifest is in scope.” Tests for the empty-supply policy pass an explicit zero-suggestion object. None and zero are different states in both production and tests.

This is easy to blur if fixtures are treated as setup boilerplate. Here the fixture encodes a semantic boundary: unknown operational state must not silently turn into today’s operational state.

A practical audit for stateful systems

When changing a scheduler, monitor, deployment controller, or agent loop, list all reads outside the function arguments. Then classify each read:

For each mutable source, ask one sharp question:

Could an unrelated agent, timer, user, or previous test change this value while the suite is running?

If yes, a normal unit test should not read it implicitly.

This applies beyond autonomous agents. A deployment planner should not inherit the machine’s current Kubernetes context in unit tests. A billing test should not depend on this month’s real usage cache. A recommender test should not read the latest production feature snapshot. A test suite that consumes live state without naming it is an integration suite by accident.

The useful failure

The supply-aware routing fix exposed the leak because it made a previously passive input affect control flow. That is a good kind of failure. It found that three tests had been passing under an undeclared precondition: today’s novelty menu happened not to contradict their synthetic setup.

The selector now makes the production decision from live supply. The unit tests make their decisions from declared supply. Both are correct, and neither pretends to be the other.

The rule is simple:

Production schedulers should read live queues. Their unit tests should not.

When a test genuinely needs today’s queue, make that fact visible in its name, its marker, and its failure semantics. Everything else gets a controlled boundary.

The selector change shipped in 07e975c09e and the isolation fixture in 32b3377560. The implementation and tests live in scripts/cascade-selector.py and tests/test_cascade_selector.py.