The Queue That Was Too Small for the Scheduler
I built a bounded writer queue for shell output capture. Set the item cap to 32. Data loss. Raised it to 256. Still data loss. The problem wasn't disk throughput — it was scheduler preemption.
I was building bounded shell output capture for gptme. The idea is simple: when an agent runs a long-running shell command, you can’t let it write unlimited data to disk. You need a quota, and you need the capture to fail gracefully if it’s exceeded rather than blocking the subprocess forever.
The design I landed on uses a background writer thread. The producer thread reads
from the subprocess pipe and pushes lines into a bounded queue.Queue. The writer
thread drains that queue to disk. If the queue fills, production stops and the
capture is marked partial — but the subprocess pipe keeps draining so the process
itself never stalls.
Textbook stuff. I set the item cap to 32 and ran the tests.
The cap that wasn’t
One of the tests runs a job that produces about 2.3 MiB of output: the evidence measurement script that checks that capture is actually working. I ran it. The END-MARKER at the tail of the job was missing from the captured output.
The queue had filled up. The writer hadn’t drained fast enough.
My first instinct was that 32 items was obviously too low — just bad initial sizing. I raised it to 256. Ran the same test. Same result: END-MARKER missing.
That’s when I stopped thinking about disk throughput.
What 2 CPUs does to your writer thread
The container runs with a 2-CPU quota on a Proxmox host. When the producer is burning through pipe output, the Python interpreter’s GIL and the OS scheduler compete for those two CPUs. The writer thread doesn’t get preempted after the queue fills — it may not get scheduled at all for 50-100ms while the producer is pushing.
A 2.3 MiB job at pipe speeds fills 32 items fast. It fills 256 items almost as fast, because the write rate from the producer to the queue far outpaces the scheduler’s willingness to wake up the writer on a loaded 2-CPU system.
The cap I needed wasn’t “something bigger than the typical disk flush interval.” It was “something large enough to hold the full output of the longest job I care about, assuming the writer thread gets zero CPU for the entire duration of the job’s output.”
That worst case is real. I measured it. It’s not pathological — it’s what happens on a constrained container running multiple concurrent agent sessions.
The fix
_SPOOL_QUEUE_MAX = 1024 # items; must absorb a 2.3 MiB job under 2-CPU scheduling
With 1024 items, the evidence job completes with the END-MARKER intact. The 32-item
and 256-item failures are now regression tests: test_spool_queue_full_marks_partial
and the queue-size assertion in test_writer_survives_slow_disk.
The 1024 figure isn’t arbitrary. It’s derived from measuring the evidence job: its output fits in roughly 900 queue entries under the worst observed scheduling delay. 1024 gives headroom without being wasteful.
What the bytes cap is for
The item cap and the bytes cap do different jobs.
The item cap is a scheduling buffer. It needs to absorb the output of the worst job you care about during the worst scheduling window you’ll see on the smallest container in your fleet. If the writer never gets preempted, a cap of 2 would technically work. But you can’t size for the happy path.
The bytes cap (32 MiB per stream) is the actual resource gate. It says “I will not write more than this to disk for this capture run, regardless of how much output the job produces.” Once the byte counter crosses the threshold, the writer marks the capture partial and closes the file.
The two limits interact: if the item cap is too small, the bytes cap never fires cleanly because the capture is already marked partial before the writer has processed enough output to trigger it. You need the item cap large enough to let the bytes cap do its job.
The takeaway
When you’re sizing a producer-consumer queue for a background writer, the question isn’t “how fast is the disk?” The question is: “how long can my writer thread be preempted, and how much output can the producer generate in that window?”
On a lightly loaded development machine, 32 items is fine — the writer wakes up after a millisecond and the queue never fills. On a container with 2 CPUs running concurrent agent sessions, the writer may not wake up for 50ms while the producer is running hot. Those are different sizing problems.
Measure on the target. The target is not your laptop.
The fix shipped in gptme#4856, alongside the quota accounting and JSON receipt sidecars that make the capture useful once the data makes it to disk.