The ack is part of the durability contract

A database method called force_commit() returned success before the transaction committed.

That sentence contains the whole bug. It also explains why the bug stayed hidden: the old SQLite journal mode accidentally made the lie hard to observe.

The race surfaced while ActivityWatch’s Rust server was moving from SQLite’s rollback journal to write-ahead logging. WAL was the performance change. It made commits cheaper and stopped readers from blocking on the writer. Then an Ubuntu CI runner started failing a datastore reload test.

The tempting conclusion was that WAL was flaky. It was not. WAL removed incidental serialization that had been propping up an incorrect acknowledgment contract.

The sequence that lied

ActivityWatch’s datastore has a worker thread. Requests arrive over a channel, the worker batches writes in a transaction, and normal watcher heartbeats are acknowledged immediately so they do not wait for a batch commit.

ForceCommit and Close are different. Their names promise a boundary:

caller sends ForceCommit
worker commits current transaction
caller receives success

The old implementation did this instead:

caller sends ForceCommit
worker marks the transaction for commit
worker sends success
worker leaves the request loop
worker commits the transaction

Most of the time, that ordering looked fine. The gap between acknowledgment and commit was small. But the caller was allowed to act during that gap because it had already been told the operation was complete.

The failing test did exactly that: call force_commit(), reopen the datastore, and populate bucket metadata from the reopened connection. Under WAL, the second reader could take a valid snapshot while the writer’s transaction was still uncommitted. The snapshot was consistent — and stale.

That distinction matters. SQLite did not return corrupt data. It honored its isolation model. The application had invented a stronger guarantee and then failed to implement it.

Why the old journal mode masked it

Rollback journals and WAL have different reader/writer behavior.

With the rollback journal, locking often serialized the reopened reader behind the pending commit. That made the sequence appear to be:

ack -> reopen -> observe committed data

But the real application ordering was still:

ack -> reopen races commit

The filesystem and SQLite lock path happened to make one outcome common. WAL changed the lock choreography: readers no longer need to wait for the writer to finish its commit. They can keep reading from a stable snapshot while the writer appends to the WAL.

That is one of WAL’s central benefits. Here it also acted as a concurrency test. Removing contention exposed an API contract that had silently depended on contention.

The macOS runner happened not to expose the race in the observed runs. Ubuntu did. Neither platform was wrong. A timing-dependent API was wrong.

The fix was to move one response boundary

The worker now defers the acknowledgments for ForceCommit and Close until tx.commit() returns:

handle ForceCommit or Close
hold response sender
leave request loop
commit transaction
send success, or propagate the commit error

Other commands are still acknowledged immediately. A heartbeat should not block for up to the 15-second batch interval. The fix did not turn every write into a synchronous write; it repaired the two methods whose public meaning already required one.

That scope is important. “Make acknowledgments durable” sounds like a reason to put an fsync on every request. That would destroy the batching design. The correct rule is narrower:

An acknowledgment must happen at the point promised by that operation’s contract.

For a queued heartbeat, success can mean “accepted by the worker.” For ForceCommit, success has to mean “the transaction commit returned.” For Close, success has to mean “the final commit returned and shutdown can proceed.” Same channel, different completion semantics.

Commit failures now travel back through those deferred responses instead of becoming silent success. That is as important as fixing the happy-path order. A durability barrier that reports success after SQLITE_FULL is worse than no barrier because it teaches callers to discard their recovery options.

WAL still needed an explicit durability choice

The migration used synchronous=FULL, not the common WAL + NORMAL recipe.

That choice makes sense only in the context of the application’s write policy. ActivityWatch already groups writes into one transaction committed every 15 seconds, after more than 100 events, or when an operation requests a commit. The fsync is paid per batch, not per heartbeat. In the measured production-sized test, WAL with FULL still improved heartbeat throughput by about 4%, from 5,572 to 5,815 heartbeats per second.

NORMAL would remove the per-commit durability point and rely on checkpoints for syncing. SQLite’s automatic checkpoint is page-count based, not time based. At ActivityWatch’s low steady write rate, a power loss could then discard much more than the intended batch window. Adding a timed checkpoint to restore the bound would mostly recreate the sync schedule already provided by batched commits plus FULL.

So the useful optimization was not “turn durability down.” It was:

• replace rollback-journal create/delete churn with sequential WAL appends;
• preserve the existing commit durability boundary;
• let readers proceed concurrently;
• make the application acknowledgment match that boundary.

That is a cleaner performance win because it removes work without weakening the promise.

A migration checklist for hidden synchronization

When a storage change reduces locking or adds concurrency, I now want four explicit checks:

1. List operations with barrier semantics. Names like flush, commit, close, checkpoint, drain, and sync are obvious suspects.
2. Locate the response before the storage boundary. If success is sent before commit, rename the operation or move the response.
3. Test an immediate dependent action. Reopen, read from another connection, terminate the process, or inspect the durable artifact immediately after the method returns. Do not add a sleep; sleeps hide the contract violation.
4. Inject boundary failure. Disk full, commit failure, or closed channel must not be reported as success by a barrier method.

This applies beyond SQLite. Replacing a global mutex with finer-grained locking, moving writes behind a queue, enabling asynchronous fsync, changing a message broker’s ack mode, or parallelizing readers can all expose the same class of bug. Old contention is often doing undocumented coordination work.

Faster concurrency does not create the race. It makes the pre-existing race reachable.

What I did not conclude

I did not conclude that WAL is unsafe. WAL behaved correctly and made a bad assumption observable.

I did not conclude that every acknowledgment must wait for disk. That would confuse queue acceptance with durability and erase useful batching.

I did not conclude that passing stress tests proves the order. The reload test passed 50 out of 50 stress iterations after the fix, which is useful evidence. The stronger proof is structural: the response sender is held until commit() returns, and the error path responds with the commit failure.

The broader lesson is simple:

The response line is part of the storage design.

We tend to review transaction boundaries, journal modes, fsync policy, and lock behavior as the “database” layer. But callers experience none of those directly. They experience the moment the API says “done.” If that moment is early, the system’s durability guarantee is early too — regardless of what the database eventually does.

Sources

• ActivityWatch/aw-server-rust#616 — WAL migration, race analysis, benchmarks, and verification
• Merge commit: d7349c2c7d1d4f990a5d7a5abbeede801104f1c9
• Follow-up disposition: the internal research note from 2026-09-05 on the ActivityWatch disk-I/O profiler idea