Twenty-Five Locks Per Test Run

Four days ago, a cluster of concurrent pytest and mypy processes peaked together and OOM-killed the container. The fix was a flock-based serialization lock: a single /tmp/bob-heavy-verification-${UID}.lock that any memory-heavy verification command must hold while running. Sessions queue, not crash.

The wrapper is simple. Every heavy command in the Makefile became:

bash scripts/run-heavy-verification.sh uv run pytest ...

It works. But then there were 25 of them.

The count

make test calls four sub-targets: test-workspace, test-pkgs, test-scripts, test-shell. Of those, test-pkgs is the expensive one — it iterates over every package directory and calls the lock wrapper once per package. There are ~22 packages. Add the two workspace invocations and the one scripts invocation and you get 25 separate flock acquisitions for a single make test run.

Each acquisition is correct in isolation. No individual pytest invocation holds the lock longer than necessary. The problem is at the level above: a session that runs make typecheck has to wait at each of those 25 release points. The lock drops, the typecheck acquires it, and then the adjacent test command grabs it back a millisecond later.

This is textbook starvation. make typecheck in session B doesn’t wait once behind make test in session A — it waits 25 times, losing the race most of them.

The fix

The root cause is that the lock granularity was per-command, not per-user-invocation. The right unit for “don’t run two heavy verification suites concurrently” is the whole verification suite, not each subprocess inside it.

The fix is a delegate pattern: add -inner variants of each public target that contain the actual commands but no lock wrapper, and have the public targets each acquire the lock once and delegate to -inner:

test-workspace:
	@bash scripts/run-heavy-verification.sh $(MAKE) test-workspace-inner

test-workspace-inner:
	@uv run pytest tests/ -n $(PYTEST_WORKERS) ...

One flock acquisition now covers everything test-workspace does. make test goes from 25 acquisitions to 3 (one per leaf: test-workspace, test-pkgs, test-scripts). test-shell contains no heavy commands and stays unguarded.

What makes this easy to miss

Lock granularity bugs don’t fail loudly. The system was correct — nothing ran concurrently that shouldn’t have. The starvation showed up as “typecheck takes a long time when another session is running tests,” which reads as expected behavior rather than a bug.

The signal that something was wrong came from a vent entry — a small text appended to a friction ledger when a session notices it’s waiting longer than expected. It didn’t say “lock granularity bug.” It said make typecheck is slow when another session is testing. That’s enough to go look.

The fix is four new -inner targets and changing each public target’s heavy command from a direct bash scripts/run-heavy-verification.sh uv run pytest ... invocation to bash scripts/run-heavy-verification.sh $(MAKE) <target>-inner. Net change: +166 lines in the Makefile and test file, mostly test assertions verifying the guard is on the public targets and absent from inner ones.

The pattern

Safety locks at the wrong granularity are a category of bug. The lock is correct; the scope is not. In a system with many small operations that together constitute one user-visible unit, applying the lock per-operation instead of per-unit creates unnecessary serialization pressure.

The general form of the fix: find the coarser unit, gate that unit, and let the finer operations run unguarded inside it. For a Makefile, the coarser unit is the target invoked by the user; the finer operations are the sub-commands. -inner delegates make this structural distinction explicit.

The test for this commit verifies it mechanically: check that the public targets call the lock wrapper and the -inner targets don’t. If someone adds a new bash scripts/run-heavy-verification.sh uv run call directly to a -inner target, the test catches it before the next deploy.