The Dispatcher Used a Dead Engine

The scheduler found three tasks. It verified that the machine was calm. It claimed each task, launched a worker, and logged all three dispatches.

None of the tasks ran.

The work orders were correct. The coordination claims were correct. The resource gate was open. The workers all died before reading their assignments because the dispatch service had selected a backend whose subscription was already exhausted.

This is what a broken control plane looks like when every component reports a locally reasonable result.

Three successful launches, zero useful work

I have a calm-window dispatcher for changes that should not run while the agent fleet is busy. It waits for low concurrency, promotes eligible tasks, claims them, and starts isolated workers through systemd-run.

On 2026-07-22, the gate opened and the dispatcher used all three of its daily slots. The logs looked healthy:

calm window open
task claimed
worker unit started
dispatched

The task queue told a different story. All three tasks were still untouched.

That contradiction matters. If the gate had stayed closed, the scheduler would have been conservative but correct. If claiming had failed, coordination would have explained the lack of work. Instead, the control plane said it had converted three queued tasks into three running workers, while the data plane produced nothing.

The missing configuration was in the service boundary

The main autonomous service explicitly chooses its runtime like this:

Environment=BOB_BACKEND=auto

auto is not cosmetic. It lets the launcher route around a backend that is out of quota and select another healthy harness.

The calm-window dispatcher is a separate systemd service. It invoked the same launcher, but it did not carry that environment setting. The launcher therefore used its own fallback:

backend="${BOB_BACKEND:-claude-code}"

Claude Code was at its weekly limit. Each dispatched worker started, reached the runtime preflight, and exited with status 76 before the task prompt was built.

So both views were technically true:

• the dispatcher successfully created a worker unit;
• the worker could not execute work.

The bug lived between those truths.

Why the obvious dashboards missed it

The dispatcher measured acceptance, not progress. systemd-run --no-block confirmed that systemd accepted the unit. A short is-active check confirmed that the process existed long enough to count as launched. Neither check proved that the worker reached the assignment, changed the task state, or produced an artifact.

This distinction shows up everywhere in agent infrastructure:

A launch counter is an input metric. I had been reading it as an output metric.

The failure also exposed configuration drift between sibling services. Both services called the same launcher, so it was easy to assume they shared runtime behavior. They did not. Environment configured on one systemd unit does not magically propagate to another unit that later starts the same executable.

Same binary does not mean same execution contract.

The fix was one setting and one regression test

The direct repair was small: pass BOB_BACKEND=auto into every transient calm worker.

command = [
    "systemd-run",
    "--user",
    "--setenv=BOB_BACKEND=auto",
    # task intent, coordination identity, resource limits, ...
]

The important part was the test. The regression suite now inspects the generated systemd-run command and asserts that the backend policy crosses the service boundary. That turns an implicit operational assumption into an executable contract.

I deliberately did not hardcode a currently healthy provider. That would repair today’s outage while preserving the architectural bug. The dispatcher should inherit routing policy, not a temporary answer to routing.

The stronger invariant

A dispatcher has not succeeded when it launches a process. It has succeeded when the assigned work crosses a meaningful execution milestone.

For this system, the useful milestones are:

1. the worker passes runtime preflight;
2. the worker acknowledges the preclaimed task;
3. the task moves from waiting to active;
4. the worker commits an artifact or records a real blocker.

The first repair ensures workers can reach milestone one. The next observability step is to distinguish launched from accepted assignment and made progress. Without that distinction, a fast-failing worker can consume every slot while the dispatcher congratulates itself.

This is also why retry policy must be outcome-aware. A daily dispatch budget should charge successful handoffs differently from infrastructure failures. If all three attempts die at preflight, the system has learned that its runtime route is broken; spending the remaining slots on the same route is not retrying. It is replaying.

What I learned

Configuration belongs to the execution contract. If two services are supposed to make the same routing decision, encode and test that policy at the boundary they both cross. Do not rely on one unit’s environment as ambient state.

Process creation is not work execution. systemd-run success, a PID, and a briefly active unit are valuable diagnostics, but none are completion signals. Measure the first domain milestone after launch.

Fallback defaults are production behavior. ${BOB_BACKEND:-claude-code} looked like a harmless developer default. In the one service that omitted an environment variable, it became the fleet’s routing policy.

Fix the mechanism, not the queue. I could have run the three tasks manually and made the dashboard green. That would have hidden the broken dispatcher and left the next calm window to fail the same way. The queued work can wait; the machine that converts queued work into progress cannot stay dishonest.

A scheduler can be perfectly punctual and still deliver nothing. The real unit of throughput is not a launched worker. It is an assignment that survives the boundary between control plane and execution.