I Gated an LXC on the Host's Load Average

At 03:00 UTC, my autonomous agent fleet stopped spawning workers because the resource gate said the machine was CPU-pinned:

load1=22.50 > 0.90*nproc(24)=21.6 — box pinned, skipping spawn

The arithmetic was correct. The inputs were not measuring the same machine.

nproc reported the 24 CPUs assigned to my LXC. /proc/loadavg included work from the Proxmox host and its sibling containers. Worse, load average counts tasks stuck in uninterruptible I/O sleep, so an I/O-bound machine looked CPU-bound. At the moment the gate stopped the fleet, CPU utilization was about 20%, memory was 58% free, and I/O pressure—not CPU demand—was the real bottleneck.

This is a nasty monitoring failure because every individual number looks plausible. The bug only appears when you ask whether the numerator and denominator describe the same scope and the same resource.

The gate that looked sensible

My autonomous runner checks a resource gate before each fan-out burst. The original CPU axis was:

load_ratio = load1 / nproc

if load_ratio > 0.90:
    return "skip"
if load_ratio > 0.75:
    return "cap at 1"
if load_ratio > 0.60:
    return "cap at 2"

That is a common heuristic. Normalize load average by CPU count and you get a rough saturation ratio. It had also survived months of tests because the tests fed both values through environment overrides. Given load1=22.5 and nproc=24, the gate did exactly what it was written to do.

The production inputs violated the model behind the formula.

Scope mismatch: host numerator, container denominator

LXCFS can virtualize /proc/loadavg, but only when its load-average support is enabled. On this host it runs as:

/usr/bin/lxcfs /var/lib/lxcfs

There is no --enable-loadavg. The load-average view therefore tracks activity outside my container, while os.cpu_count() is constrained to my 24 assigned CPUs.

The process-count field in /proc/loadavg made the mismatch visible. It showed roughly 3,443 tasks; counting threads inside the container gave roughly 2,547. That ~896-task gap was host and sibling-container activity leaking into the numerator.

So the gate was effectively computing:

(host + Bob + Alice + Gordon + Sven load) / Bob's 24 CPUs

No threshold can repair that ratio. Tuning 0.90 to 1.20 would only make the wrong measurement fail less often.

Semantics mismatch: load is not CPU demand

Even a perfectly container-scoped load average would still be the wrong signal for this decision.

Linux load average includes runnable tasks and tasks in uninterruptible sleep, usually waiting on I/O. That is useful as a broad measure of demand, but my gate was asking a narrower question: will another agent process contend for CPU?

During the incident, I/O PSI full avg10 was around 33%. The box was waiting on storage. Treating those blocked tasks as CPU demand caused the gate to skip all new work, including work that would have spent most of its time waiting on an LLM API.

The gate needed separate signals for separate resources.

Pressure Stall Information is the right primitive

Linux Pressure Stall Information (PSI) reports how much time tasks lose because a resource is unavailable. Crucially, cgroup pressure files give me a container-relevant view:

/sys/fs/cgroup/user.slice/user-1000.slice/cpu.pressure
/sys/fs/cgroup/user.slice/user-1000.slice/io.pressure

The gate now reads some avg10 from each:

• CPU PSI: time when at least one task was ready to run but waiting for CPU.
• I/O PSI: time when at least one task was stalled on I/O.

Those become independent axes:

# CPU pressure can stop a spawn.
if cpu_psi_some >= 80:
    return "skip"
if cpu_psi_some >= 60:
    cpu_cap = 1
elif cpu_psi_some >= 40:
    cpu_cap = 2

# I/O pressure only reduces fan-out width.
if io_psi_some >= 70:
    io_cap = 1
elif io_psi_some >= 50:
    io_cap = 2

CPU pressure may skip a burst because another CPU-hungry process directly worsens CPU saturation. I/O pressure only caps the burst. Agent sessions are not uniformly disk-heavy, so high I/O pressure is evidence for caution, not evidence that all new work is harmful.

The final width is the strictest cap contributed by CPU, I/O, memory, cgroup memory, and model-quota axes. /proc/loadavg remains in the JSON output for observability, but it no longer makes decisions.

Fail open when PSI is unavailable

PSI is available on the current Linux deployment, but the script also runs in test environments and may be reused elsewhere. If neither the cgroup pressure file nor /proc/pressure/* can be read, that axis contributes no cap.

That is deliberate. An unavailable sensor is not proof of saturation. Failing closed would turn a kernel/configuration difference into a silent fleet outage. The other axes—memory, cgroup memory, and quota—continue to protect the box.

Every pressure input and threshold is environment-overridable, so the full decision table remains testable without running stress-ng or deliberately stalling a disk.

What I got wrong in the original design

I previously wrote that load average was the right alternative to process count for this gate. That argument got one distinction right: process count is not resource pressure. It missed two more important distinctions:

1. Metric scope must match control scope. A container-local controller cannot safely normalize a host-wide numerator with a container-local denominator.
2. Metric semantics must match the intervention. A spawn gate deciding CPU contention needs CPU stall time, not a combined runnable-plus-I/O-wait queue.

This is the broader lesson for automated control loops. A dashboard can tolerate an ambiguous metric because a human brings context. A gate turns the metric into action. Once a number can stop production, “roughly correlated” is not enough: scope, units, and causal relationship all have to line up.

The fix landed in commit 4b6fdbc620, with regression tests for high CPU PSI, moderate CPU PSI, high I/O PSI, combined caps, missing PSI files, and the exact case that exposed the bug: absurdly high load average with low CPU pressure no longer blocks the fleet.