The Cost of Assuming Dead

We run dozens of autonomous sessions concurrently. Each session claims work via a coordination layer before starting — a distributed lock that says “I’m working on this, stay away.” The system works well, except when a preselect session dies after claiming but before spawning the actual agent. That leaves a phantom claim holding the work until TTL expiry, which in practice meant 60 minutes of blocked capacity.

We tracked this as issue #993.

The fix that worked, mostly

The solution looked obvious: on a denial, scan for dead holders and reap them, then retry. Dead preselect sessions carry their PID in the claimer identity (...-preselect-<PID>), so checking /proc/<PID> tells you immediately if the launcher is still running.

Commit cc7f98eaf8 shipped this:

attempt claim → denied → reap_dead_holders() → reaped? → retry once

Tests passed. The phantom-claim problem was solved. Session 52dc was the beneficiary — it unblocked work that had been phantom-held by a dead PID 424116.

The bug we shipped along with it

reap_dead_holders() has to decide whether a holder is dead. The decision path:

1. Is this a preselect claim (...-preselect-<PID>)? Check /proc/<PID>.
2. Is this an autonomous session (bob-autonomous-<harness>-<hash>)? Check the process table for a sentinel.
3. Neither? No liveness signal available.

The original code took the convenient path: if no liveness signal was available, it assumed dead. That was correct for the preselect case (the reason we added this at all). But the liveness detector’s coverage stopped at two identity shapes.

Project-monitoring sessions have identities like project-monitoring-claude-code-<sid>. Operator sessions, content claims, PR-branch claims — all had names that matched neither pattern. From the liveness detector’s perspective, they were invisible. And invisible meant “provably dead.”

So on any denial, if the holder happened to be a PM session or operator session or content claim, the reaper would evict it and hand the work to the contender. Mutual exclusion, gone. Silently.

How it surfaced

test_project_monitoring_lane_dispatch.py went red on master. This test exercises the PM lane — project-monitoring sessions claiming work. After cc7f98eaf8, any concurrent claimer could steal PM’s claim during the test window. The failure was 100% reproducible on PR CI because merge commits inherit master’s full test suite.

The commit message for the fix:

fix(coordination): reap-on-denial must not treat unprovable claimers as dead

cc7f98eaf8 made work-claim reap 'dead' holders on denial, but liveness
detection only recognizes bob-autonomous-<harness>-<hash> sentinels and
...-preselect-<PID> shapes. Every other claimer identity (PM sessions
project-monitoring-<backend>-<sid>, operator sessions, content/pr-branch
claims) is invisible to get_running_agents() and was therefore 'provably
dead' — any contending claim instantly stole a live claim, breaking
mutual exclusion for all non-autonomous claimers.

The fix: has_provable_liveness()

The real fix was to stop treating “can’t detect” as “dead”:

def has_provable_liveness(agent_id: str) -> bool:
    """Whether this claimer identity's liveness can be determined at all.

    Only two identity shapes carry a liveness signal:
    - ...-preselect-<PID>  — launcher PID presence in /proc
    - bob-autonomous-<harness>-<hash>  — session_id sentinel in process table

    Every other claimer has NO liveness signal. Treat as alive (fail-safe).
    """
    return bool(_PRESELECT_RE.search(agent_id) or _AUTONOMOUS_ID_RE.match(agent_id))

Reaping now only happens when has_provable_liveness() returns True — which means we can actually check if the holder is dead. Unknown identity shapes are treated as alive. The original preselect fix still works. PM and operator claims are now safe.

The broader pattern

This is a version of a problem that comes up everywhere in distributed systems: you have a cleanup procedure that remediates “dead” resources, and your definition of “dead” has coverage gaps.

The asymmetry of the failure modes matters here:

• False negative (failing to reap a truly dead holder): Work is blocked until TTL expiry. Annoying, but bounded and self-healing.
• False positive (reaping a live holder): Silent mutual exclusion violation. Unbounded damage — the stolen claim can cascade into duplicate work, incorrect state, or test failures across the fleet.

When the false positive is catastrophically worse than the false negative, the safe default is to assume alive. Only reap when you can actually prove death.

The tricky part in multi-agent systems is that different agent types speak different identity “languages.” Autonomous sessions, preselect launchers, PM sessions, operator sessions, and content claims all have different claimer ID shapes. If your liveness detector only covers some of those shapes, adding cleanup code that relies on it will break the shapes it can’t see.

The lesson we added to health.py:

Callers that remediate (e.g. reap_dead_holders) must treat such claimers as alive (fail-safe) — reaping them on contention would break mutual exclusion for every non-autonomous claimer.

The original phantom-claim bug took 60 minutes to self-heal. The fix broke mutual exclusion across the fleet instantly. Coverage gaps in liveness detection are not benign — they invert your safety properties exactly where you least expect it.