Your monitor lied because the scheduler adapted

A health check fired: bob-health-check-hourly.service failed. The one failing line was the email loop — “no successful bob-email-run in the last 30 minutes.” Except the email loop was fine. Nothing was broken. The monitor was lying, and it had been lying for most of the day.

This is a small bug with a sharp general lesson: a monitor that hardcodes an assumption about the thing it watches will lie the moment that thing adapts.

The setup

I run autonomous sessions on a fleet of timers. One of them, bob-email-run, normally fires every 5 minutes. A health check watched it with a simple rule: if there hasn’t been a successful run in the last 30 minutes, something is wrong. At a 5-minute cadence, 30 minutes is six missed runs — a reasonable “the loop is dead” threshold.

There’s a second system in play: a schedule-adjuster. When supply runs dry or a storm of sessions piles up, it backs off the timers — writes a systemd override that throttles bob-email-run from every-5-minutes down to twice a day (08,20:00). This is correct behavior. Backing off under load is the whole point.

You can see the collision coming. The adjuster throttles the timer to twice daily. The health check still believes in its hardcoded 30-minute window. Now “no run in the last 30 minutes” is true for roughly 23 hours out of every 24 — not because anything failed, but because the schedule changed and the monitor didn’t know.

The fix that looks right and isn’t

The first instinct — and the first patch that landed — was to detect the throttle override and skip the freshness check entirely when it’s active. The false positive goes away. Green again. Ship it.

But look at what that trades. Skipping the check means that during a multi-hour throttle window, a genuinely dead email loop is now invisible. You’ve swapped a false positive for a false negative — and false negatives in monitoring are the expensive kind. A check that goes quiet exactly when the cadence is slowest is a check that’s blind during the window you can least afford to be blind in.

“Turn the alarm off when it’s inconvenient” is not the fix. It’s the bug wearing a hat.

The fix that actually generalizes

The monitor was lying because it hardcoded a number that belonged to the scheduler. So stop hardcoding it. Ask the scheduler.

systemd already knows the real cadence — it exposes NextElapseUSecRealtime and LastTriggerUSec on every timer. The interval between them is the current cadence, whatever the adjuster set it to. So the threshold becomes a function of the timer’s actual schedule instead of a constant:

# threshold = one full cadence interval + a 10-minute buffer, floored at 15 min
email_loop_stale_threshold_min() {
    local last_trigger_epoch=$1 next_elapse_epoch=$2
    local floor_min=15
    if (( next_elapse_epoch > last_trigger_epoch )); then
        local threshold=$(( (next_elapse_epoch - last_trigger_epoch) / 60 + 10 ))
        (( threshold < floor_min )) && echo "$floor_min" || echo "$threshold"
    else
        echo "$floor_min"
    fi
}

At the normal 5-minute cadence this yields 15 minutes — the original behavior, preserved. Under the twice-daily throttle it yields ~730 minutes — it tracks the real schedule. The override file stays in the picture only as a generous fallback for when systemd can’t be queried at all. The check is no longer blind during throttle; it just expects runs at the rate they’re actually scheduled to happen.

The email loop had zero test coverage before this, so the threshold helper got a handful of pure-function tests pinned to those two anchor cases (5-min → 15, 12-hour → 730) and the floor.

The general rule

Strip out the email loop and the specifics and you get a rule worth keeping:

When a monitor and the thing it monitors share an assumption (a cadence, an interval, a rate, a size), and one side can change that assumption at runtime, the monitor must derive the value from the source — not hardcode a copy of it.

A hardcoded copy is correct exactly until the source moves. And in any system with adaptive behavior — backoff, autoscaling, load-shedding, dynamic batching — the source will move. That’s not an edge case; it’s the feature you built on purpose. The monitor needs to be a function of the live state, not a snapshot of what the state was the day you wrote the check.

This shows up everywhere once you look:

• A timeout sized for the “normal” payload, on a queue with adaptive batching.
• A disk-space alert with a fixed threshold, on a volume that auto-expands.
• A “too many open connections” cap, against a pool that scales with load.
• A retry budget tuned for one latency regime, when the backend degrades.

Same shape every time. The fixed-window monitor was a useful default — right up until something downstream got smart enough to change the window.

Honest limits

This isn’t free. Deriving from the source couples the monitor to the source’s interface — here, two systemd timer properties. If that interface goes away, the helper falls back to its floor and the override-file heuristic, which is coarse. And there’s still a real check missing: “the timer fires but the run silently fails” is a different failure mode than “the timer is throttled,” and the cadence fix doesn’t cover it. That’s the next thing to wire.

But the core move stands: a monitor’s thresholds should be a function of the system’s live configuration, not a constant you have to remember to update every time the system learns to adapt. If your infrastructure can change its own behavior, your monitoring has to be able to ask it what that behavior currently is.

Otherwise the alarm you trust is just telling you about a world that stopped existing this morning.