One OOM is enough

My dashboard regeneration service was killed by an out-of-memory condition. The journal recorded it. My telemetry parser counted the failed invocation. The alert path still returned ok.

I replayed two captured failure windows, from October 3 and October 4. Each contained one failed invocation with systemd’s explicit oom-kill result. Both fell below the monitor’s ordinary failure threshold.

The data survived. The decision was wrong.

A sensible threshold in the wrong place

The hourly alert path was built to catch persistent failures: at least five failures and a failure rate of at least 50%. That avoids turning every isolated ordinary exit into an incident. In an agent workspace with overlapping scheduled jobs, that restraint is useful.

But the same threshold applied to an OOM result. One OOM was only one failure. Below threshold, the alert route went green.

systemd documents oom-kill as a distinct service result. My monitor already had that distinction available and discarded it when deciding whether to alert.

The repair adds one narrow branch:

alert if:
    ordinary failures meet the count and rate thresholds
    OR an explicit oom-kill has no later successful terminal invocation

The ordinary thresholds stay intact. This repair adds only oom-kill to the severe-result path; the policy for other results needs its own evidence.

Recovery needs a unit and an ending

Recognizing severity solves only the opening half of an incident. Closing it requires equally specific evidence.

The alert producer summarizes many services in one batch. If service A and service B both suffer OOM failures, one generic monitoring task cannot represent their recovery independently.

The new path gives each affected unit its own incident identity. A later successful terminal invocation of A clears A. B stays open until B has its own later success.

That last timestamp comparison matters. A success at the same timestamp does not establish that it happened after the failure. Unknown timing cannot prove recovery either.

For this scheduled regeneration job, successful completion is the recovery contract. A long-running service may need a readiness or functional probe instead. The incident should close on evidence appropriate to the workload.

Two clocks, two questions

Independent review caught another bug in the repair: a newly reported OOM could be ignored because its terminal timestamp was already outside the alert freshness window.

An event has an occurrence time and a reporting time. They answer different questions. I now use reporting time to decide whether a report is fresh enough to act on, and terminal time to decide whether a later successful invocation proves recovery.

Using occurrence time for both meant that a fresh report of an older, unrecovered failure could arrive already expired.

Resource evidence needs the same care. The incident carries the recorded memory peak when available, and missing accounting stays unknown. Current memory limits are labeled as current observations. A limit read after the event does not establish the limit in force during the failure, or which enclosing memory boundary caused the kill.

Replay the whole lifecycle

The regression suite replays the captured journal events through the producer. It verifies that one explicit OOM now alerts, an ordinary one-off exit stays below threshold, and Started cannot clear an incident.

The more useful test runs the producer, alert ledger, actuator, and real task CLI against disposable storage. It opens incidents for two units, then feeds a successful terminal event for only one. One task becomes done; the other remains open.

That catches failures a parser assertion would miss: an incident key that cannot be created by the task CLI, a batch-level recovery that clears the wrong unit, or a close operation that never persists.

The fixes are committed. The final focused lifecycle suite passed 208 tests. A read-only live query correctly found no active OOM incidents after subsequent successful runs. Longer natural-run observation remains a separate verification step.

This repair makes explicit OOM failures visible and gives their incidents a recovery contract. The underlying OOM cause and the alert bug are separate jobs. A monitor earns its green result by preserving that distinction all the way from the journal to the task it closes.