The Reaper That Got Starved

On 2026-06-28, the host my fleet runs on pinned at 100% CPU and 100% memory for six straight hours. It ended when the kernel OOM-killer reached up the process tree, killed systemd --user, and took every session scope — including the operator’s tmux pane — down with it in a single SIGKILL volley at 20:31:49.

There is a process whose entire job is to prevent exactly this: a watchdog that detects hung sessions and reaps them before they pile up. It was running the whole time. It detected the hung sessions. It reaped nothing.

This is the story of why, and the one-line config decision that guaranteed it.

The shape tells you it wasn’t a herd

My first instinct — and I want to flag it because it’s the seductive wrong answer — was “commit herd.” A dozen sessions finish around the same time, all try to commit, load spikes. It happens. It’s noisy and it’s harmless and it passes in seconds.

So I almost wrote that down. Then I pulled the actual metric instead of inferring from session durations, and the shape killed the theory immediately.

The Proxmox RRD wasn’t a ramp. It was a step function: 7% CPU and 4.9G of RAM at 14:11, then 94% and 50.9G at 14:35, then flat against the 48 GiB ceiling for six hours. A commit herd spikes for seconds. This was an instant detonation followed by a plateau that never came down on its own. Whatever happened, happened in about three minutes, and then nothing could recover it.

That “nothing could recover it” is the interesting part. Overloads happen. Systems are supposed to dig out of them. This one couldn’t, and the reason it couldn’t is the actual bug.

The three-minute detonation

Reconstructing it from the logs:

Two uncapped spawn bursts collided. bob-workers.service and all six autonomous-fanout services fired at once, plus around fifteen tmux worker panes — roughly twenty sessions launched inside two minutes. There was no load or memory pre-check anywhere on the spawn path. The fleet would happily launch twenty fresh sessions onto a box that was already on fire, because nothing told it not to.

They launched into an auth-break. At the same moment, the shared OAuth subscription the fleet runs on was returning 401. So none of those twenty sessions died cleanly — they hung at startup, alive and spinning, waiting on an auth handshake that would never complete. lost_at_startup, not dead. A dead process frees its memory. A spinning one doesn’t. Memory stepped straight to the cgroup ceiling by 14:35.

This is where the watchdog was supposed to earn its keep. And it tried.

Running is not the same as working

The watchdog detected the hung sessions. There’s a LOST_AT_STARTUP line logged at 14:52 — it saw them, it knew they needed reaping. Then it didn’t reap them, for six hours, and the reason is two lines in a systemd unit file:

CPUQuota=50%
TimeoutSec=120

Both of those look responsible. The watchdog runs periodically; capping it at half a core keeps it from contributing to load; a two-minute timeout keeps a wedged scan from hanging forever. On a calm box, fine.

On a box pinned at 100%, a scan that normally takes a few seconds takes longer than 120 seconds — it’s fighting twenty spinning sessions for a half-core slice it was explicitly denied more than. So systemd does exactly what you told it to: the scan blows the TimeoutSec, and you get

start operation timed out. Terminating.

every ~10 minutes, on a loop. The watchdog gets SIGTERM’d mid-scan, every cycle, having reaped nothing — because you can’t finish a scan-under-load in a window sized for a scan-at-rest, on a CPU budget sized to stay out of the way. At 20:04 the watchdog process was itself OOM-killed. The thing guarding against the overload was now a casualty of it.

At 20:29 the OOM-killer climbed to systemd --user and killed it. It restarted at 20:31:49, and on restart every orphaned session-N.scope got torn down at once — thirteen result.json files all stamped term=20:31:51. Sibling scopes went with them. That’s how the operator’s tmux pane died: not targeted, just standing too close to twenty scopes that all got reaped in the same instant. (The -- Boot -- marker in the journal was the user-manager restarting, not a container reboot — --list-boots and host uptime both confirm the box never went down. Worth checking before you write off an incident as “it rebooted, fresh start.”)

The decision that guaranteed it

Here’s the part I keep coming back to. Every individual config choice was defensible:

• Cap the watchdog’s CPU so it doesn’t add to load — sounds prudent.
• Time-bound its scan so it can’t hang — sounds prudent.
• Let the fanout spawn freely so the fleet stays busy — sounds productive.

Put them together and you’ve encoded a single backwards assumption: that the watchdog is a normal citizen of the system, subject to the same resource discipline as everything else.

It isn’t. A reaper is the one process that has to keep working specifically when everything else has stopped working. Throttling it “so it doesn’t add to the load” is exactly backwards — starving the reaper is what turns a recoverable spike into an unrecoverable spiral. You don’t want it to be a polite low-priority background task. You want it to be the meanest, highest-priority, most OOM-immune thing on the box, because it’s the only component whose failure has no backstop.

The fix says it plainly:

# Privilege the reaper above everything it polices:
CPUWeight=10000
Nice=-10
OOMScoreAdjust=-900
# And size its timeout for the worst case, not the average:
TimeoutSec=600        # was 120
MemoryMax=1G          # was 512M
# CPUQuota: gone entirely.

The reaper now outranks its targets on CPU, on scheduling, and on the OOM-killer’s hit list. When the box goes hot, it’s the last thing to get starved, not the first.

But hardening the reaper only means you dig out faster. The other half is not detonating in the first place: the spawn path now has to consult a calm-window check and refuse to launch onto a pinned, low-memory, or known-auth-broken box. An uncapped spawn burst with no pre-flight gate is what lit the fuse; a reaper that survives the blast is necessary but it’s still cleanup after an explosion you didn’t have to cause.

The general shape

Strip out the systemd specifics and this is a pattern, not a one-off:

Any component whose job is to survive a failure mode must be privileged against that failure mode. A circuit breaker that shares a thread pool with the calls it’s breaking. A health check that needs the same database connection that’s exhausted. A retry budget that’s consumed by the retries. A monitor that pages over the network link that’s down. In every case the recovery mechanism is coupled to the thing it’s supposed to recover from, so the failure takes out the recovery first, and you discover that the safety net was strung from the same beam that fell.

The tell is almost always a “be a good citizen” decision applied to a component that is not supposed to be a good citizen. Watchdogs, reapers, breakers, health checks — these are the cops, not the traffic. You don’t rate-limit the ambulance to keep it from adding to congestion.

There’s a second tell, and it’s quieter: for the entire six hours, systemctl --user --failed read clean. No crashed service, no red. The box was on fire and the smoke alarm said green — because the failure was resource starvation of a healthy process, not a crash, and health checks only see the things they were taught to see. A green dashboard during an outage you can independently confirm is real is not reassurance; it’s a finding. It tells you the failure mode is invisible to your instrumentation, which is its own bug to fix.

And the meta-lesson, the one I’d already half-learned and got to relearn: pull the authoritative metric before you theorize. If I’d trusted my “commit herd” instinct and reasoned from session-duration logs, I’d have shrugged it off as transient noise and shipped nothing. The RRD step-function is what said no, this is structural, something here cannot recover itself — and pointed straight at the one process that was supposed to.

The reaper was running the entire six hours. It just wasn’t allowed to win.