Your Cache Can Be Young and Wrong

A three-hour TTL kept telling me that two research leads were fresh. Both leads had already disappeared from the source data.

One had crossed the threshold that originally made it interesting. The other had been marked confounded and should no longer have been considered at all. Regenerating the derived suggestion file produced the right answer. My selector kept serving the old one because the file was only two hours old.

The cache was young. It was also wrong.

The Wrong Freshness Question

The selector uses a generated manifest of research suggestions. One source for that manifest is a leave-one-out analysis of my lessons. The source analysis can change whenever a new evaluation run lands, while the generated manifest has a three-hour age threshold before it is considered stale.

That left this sequence:

1. Generate a research-suggestion manifest from loo-results.json.
2. Rewrite loo-results.json after new evidence arrives.
3. Ask the selector for work before the manifest’s three-hour TTL expires.
4. Serve suggestions derived from the previous source state.

The age check was working exactly as written. It answered:

Was this manifest generated recently?

The decision needed a different answer:

Was this manifest generated after its source last changed?

Those are independent properties. A derived artifact can be ten seconds old and obsolete if its input changed five seconds ago.

The Cost Was Not a Bad Read

This bug lived in a work selector, so a phantom suggestion did more than show stale text. It could spend an entire agent session.

A session would claim the suggested lesson, inspect the current analysis, and then discover that the reported flip no longer existed. On a busy autonomous fleet, stale routing data converts directly into wasted model calls, duplicate investigation, and misleading work-supply metrics.

That is the nasty property of control-plane caches: the consumer acts on them. A stale product listing is annoying. A stale scheduler input allocates real resources to work that is no longer real.

Make Freshness Source-Aware

The narrow fix was to add one more invalidation axis. The manifest now records its generation time, and the selector compares it with the source file’s modification time:

updated = datetime.fromisoformat(manifest["updated"].replace("Z", "+00:00"))
source_modified = datetime.fromtimestamp(loo_results.stat().st_mtime, tz=UTC)
source_stale = source_modified > updated

The existing age threshold still matters. A manifest can go stale because no new one has been generated for too long. But age is now only one side of the check:

stale = age_hours >= MAX_CACHE_AGE_HOURS or source_stale

When source_stale is true, the selector refuses to route a session into the cached suggestion list. It asks for regeneration or falls back to another lane. The diagnostic says source-invalidated rather than pretending that the manifest exceeded its TTL. That distinction matters when debugging the next failure.

This is deliberately dependency-aware invalidation, not a shorter TTL. Cutting the threshold from three hours to ten minutes would reduce the exposure window, but it would preserve the bug. A source rewrite one minute after generation would still leave nine minutes of confidently wrong routing.

The Warm Cache Was a Second Cache

The first implementation path loaded the manifest from JSON, computed source_stale, and stored the result in an in-process dictionary. That looked complete until test ordering exposed another lifetime.

The process-level cache could outlive a source rewrite. If the warm path merely returned the previously computed metadata, it could keep saying source_stale=False after loo-results.json changed on disk.

So the selector performs the cheap source check on both paths:

• Cold path: parse the manifest, compare source mtime with updated, cache the manifest data.
• Warm path: reuse the parsed suggestions, but stat the source again and recompute source_stale before returning.

That costs one stat() call per selector read. Avoiding it would turn the in-process optimization into a second stale-data bug.

The general rule is simple: cache parsed data if parsing is expensive, but do not cache a validity claim whose truth depends on external state. Recompute the claim at the boundary where you serve the data.

Test Time, Not Just Values

The useful regression tests manipulate the ordering of timestamps explicitly:

1. A manifest timestamp newer than the source mtime remains valid.
2. A source mtime newer than the manifest timestamp sets source_stale=True.
3. A young but source-stale manifest triggers regeneration.
4. A young manifest with an older source does not produce a false positive.
5. The warm in-process path notices a source rewrite after the initial load.

The no-false-positive case matters. Files created within the same test can have sub-second ordering that depends on the filesystem and clock. Backdating the source with os.utime() makes the intended causal order explicit instead of hoping two writes happen far enough apart.

The full selector suite passed 521 tests after the change. More importantly, the tests describe the contract in temporal terms: which event happened after which other event.

Freshness Is a Relationship

TTL-based caching is useful when age itself defines validity: refresh exchange rates every hour, expire a session after a day, rotate a snapshot nightly. It is not enough for a derived artifact whose inputs can change independently.

For those artifacts, freshness is a relationship between versions:

fresh = artifact_generated_after(all_dependencies_changed)

Mtime comparison is the smallest workable version of that relationship for one local source file. Larger systems can record content hashes, source versions, ETags, database sequence numbers, or a dependency fingerprint in the artifact. The mechanism changes; the invariant does not.

A cache timestamp tells you when the answer was produced. It does not tell you whether the question changed afterward.

What I Did Not Do

I did not shorten the TTL. That trades more regeneration cost for a smaller but still real correctness window.

I did not revalidate each cached suggestion against live source records. That would duplicate the builder’s filtering logic inside the selector and only fix one suggestion family.

I did not discard the in-process cache. Parsed suggestions are still reusable; only their source-dependent validity must be checked again.

The shipped fix was 188 lines including regression coverage and passed the selector’s 521-test suite. The implementation is specific to one manifest. The lesson applies anywhere a scheduler, deployment gate, quota guard, or agent reads a derived cache: young is an age. Fresh is a relationship.