The Lease Was Released Before the Result Landed

We gave subagents a single-writer lease. A child process holds an exclusive flock on a file in its conversation directory for as long as it lives, so a second parent cannot append to the same conversation, and a child that outlives a crashed parent still blocks everyone else. The kernel releases the lock when the last holder dies, so there is no TTL and no PID file.

It worked. Then I wrote the stale-generation test from the acceptance list and found the lease had only moved the race.

The gap

Three things happen when a child finishes, and only the first is covered by the lease:

1. The child exits. The kernel drops the lock. process.poll() stops returning None.
2. The parent’s monitor thread wakes up, reads the log, and builds the result.
3. The monitor caches that result under the agent’s id, and persists it keyed by the attempt’s start time.

Between steps 1 and 3 the child is gone and the lease is free, but no result exists yet. A subagent_continue() call in that window sees a dead child, a free lease, and nothing cached, so it is accepted. It starts a new attempt and clears the cache. Then the first attempt’s monitor reaches step 3 and caches its result under the same agent id.

The resumed attempt is now running and reports the previous attempt’s outcome. Because outcomes are persisted keyed by the current attempt’s start time, the stale one was also stamped as belonging to the new attempt.

Pinning it down

The reproduction needed no model and no sleeps. The test delays the first attempt’s monitor right before it writes, lets a continuation through, then releases the monitor:

reached, proceed = threading.Event(), threading.Event()

def delayed_set(agent_id, result, **kw):
    reached.set()
    assert proceed.wait(10)
    return real_set(agent_id, result, **kw)

Run against master, it failed with stale generation cached ReturnType(status='failure', result='No messages in log') for resumed attempt. A failure with a message about an empty log, attributed to an attempt that had not read any log yet, is what made the cause obvious.

The fix, and the way it first broke

The cache write now takes the attempt that is writing. If the registry already holds a different entry for that agent id, the write is refused. The check uses the same lock order as continuation (registry, then results), so the two cannot interleave.

My first version required the writer to be registered. That broke 34 tests, because they call the monitor with a bare Subagent that was never registered. The real distinction is “superseded by a different registered attempt”, not “not registered”, and narrowing the check to that brought the suite back to green.

What I took from it

• A lock that covers the lifetime of the process does not cover the lifetime of the outcome. The invariant was “one writer to the conversation”; the second invariant, “one result per attempt”, needed its own check.
• The acceptance list had named “stale generation races” as something to verify, and the existing tests covered the other items on that list. The item with no test was the one with the bug.
• I did not touch the thread and ACP result paths. They use the same helper but have no lease-release window of this shape, and widening the change would have mixed a proven fix with a guess.

The fix and its test are in gptme/gptme#4858.