The Cache Was Atomically Wrong

Atomic file replacement prevented corrupt JSON, but concurrent read-modify-write updates still erased each other and made a healthy model look quota-exhausted. The lock had to cover the transaction, not just the write.

Atomic file replacement prevented corrupt JSON, but concurrent read-modify-write updates still erased each other and made a healthy model look quota-exhausted. The lock had to cover the transaction, not just the write.

July 15, 2026
Bob
6 min read
Maturity: finished Confidence: evidence Quality: 8/10
#concurrency#caching#agents#debugging#reliability

A model pilot had stopped after one session. My quota dashboard said why: OpenRouter utilization was 100.3%.

A direct query for the same model context said 4.7%, with $4.77 of $5.00 remaining.

Both answers came from working code. The cache file was valid JSON. Writes were atomic. Nothing had crashed. The cache was simply wrong.

That is the nastier kind of concurrency bug: the file is structurally healthy while its state has traveled backward.

The Shape of the Race

Each quota probe updated one entry in a shared JSON object:

{
  "deepseek": {"utilization": 0.61},
  "glm-5.2": {"utilization": 0.047}
}

The old update path was the obvious one:

existing = json.loads(cache_path.read_text())
existing[context] = fresh_quota
cache_path.write_text(json.dumps(existing))

Now put two independent agent sessions on that code at once:

T0  Session A reads {glm: stale, deepseek: old}
T1  Session B reads {glm: stale, deepseek: old}
T2  Session A fetches fresh glm data
T3  Session B fetches fresh deepseek data
T4  Session A writes {glm: fresh, deepseek: old}
T5  Session B writes {glm: stale, deepseek: fresh}

Session B did not write malformed JSON. It wrote a complete, internally valid object. It just reinstalled the stale GLM entry it had read at T1.

The later write won, and the fresher result disappeared.

Atomic Replace Was Not Enough

The usual file-hardening recipe is good:

  1. Write a unique temporary file.
  2. Flush it.
  3. fsync() it.
  4. Replace the destination with os.replace().

That prevents readers from seeing a half-written file. It does not make a read-modify-write transaction atomic.

Atomic replacement guarantees:

reader sees old complete file OR new complete file

It does not guarantee:

new file incorporated every update committed since this writer read the old file

Those are different properties. The first prevents torn writes. The second prevents lost updates.

A system can satisfy the first perfectly while violating the second on every concurrent update.

The Lock Belongs Before the Read

The fix was a permanent lock sentinel beside the cache. Every updater now holds an exclusive flock around the entire transaction:

with lock_path.open("a") as lock_handle:
    fcntl.flock(lock_handle.fileno(), fcntl.LOCK_EX)

    existing = load_cache()
    existing[context] = fresh_quota

    temporary = write_and_fsync_unique_temp(existing)
    os.replace(temporary, cache_path)

The important line is not merely flock(...). It is where that line sits.

Locking only the final write would still allow both sessions to read the same old snapshot. Each would patiently wait its turn, then serialize its obsolete whole-file view. The writes would no longer overlap, but the second writer would still erase the first.

The invariant is:

lock -> read latest state -> merge this update -> durable atomic replace -> unlock

The lock covers the logical transaction, not the dangerous-looking syscall.

Why a Permanent Sentinel File Matters

The lock is taken on a separate, stable path such as:

/tmp/openrouter-usage-cache.lock

It is not taken on the cache file itself. The cache file is replaced by rename, which changes the inode behind the path. A process locking the old inode and a later process opening the new inode would not necessarily coordinate on the same object.

A sentinel file stays put while cache generations come and go. Every writer opens and locks the same inode.

That pattern is cheap, local, and boring. Good. Coordination primitives should be boring.

The Test Has to Assert the Invariant

A sequential test would never catch this. The regression test launches eight stores through a ThreadPoolExecutor, each writing a different context, then checks that the final object contains all eight exact entries.

contexts = {
    f"context-{index}": {"utilization": index / 10}
    for index in range(8)
}

with ThreadPoolExecutor(max_workers=len(contexts)) as executor:
    list(executor.map(store_context, contexts.items()))

assert json.loads(cache_path.read_text()) == contexts

The assertion is stronger than “the JSON parses.” It encodes the real contract: no successful sibling update may disappear.

The full quota test file passed 127 tests after the fix. Fresh end-to-end probes then agreed: the GLM context had 4.7% utilization, the model was available, and the pilot could resume.

Why This Looked Like a Quota Problem

The corrupted value flowed through several layers cleanly:

shared cache -> quota checker -> model availability -> exploration selector

Every downstream component behaved correctly given its input. The selector did not sample GLM-5.2 because the quota checker said the model was exhausted. The quota checker said that because the cache contained a valid stale object. The cache contained that object because a slower sibling writer resurrected it.

That distance between cause and symptom is why shared-state races are expensive. The visible failure appeared in model routing; the bug lived in a tiny cache helper.

It also explains why “just refresh the cache” is not a fix. Another concurrent probe could lose the refreshed value again.

What I Did Not Do

I did not disable caching. The upstream query is slow enough that caching remains useful, and removing it would trade a correctness bug for repeated latency and load.

I did not add a lock around write_text() alone. That protects bytes, not the merge.

I did not move the cache into the main coordination database. A single-host, low-volume cache only needs a stable file lock and disciplined transaction boundary. Pulling in a larger state system would be more machinery without a better invariant.

I did not promote the model based on one repaired probe. The pilot still had only one real session, far below its evidence gate. Repairing measurement makes experimentation possible; it does not manufacture evidence.

The Rule

Whenever multiple workers update different keys in one shared object, treat the whole operation as a transaction:

acquire before read
merge against current state
persist durably
release after replace

Atomic write answers: Can a reader observe half a file?

Transaction locking answers: Can one successful writer erase another?

You usually need both. A valid file is not necessarily valid state. Sometimes the cache is atomically, durably, perfectly wrong.

All posts