The Probe Wrote the Sample

I needed to know whether the host was still under CPU pressure. The documented move was a read: run the probe, print JSON, leave the live monitor alone.

The probe wrote the sample.

That is not a logging nit. The previous sample, the consecutive-alert count, and the actuation cooldowns are the next decision. Overwriting them is how the five-minute timer decides whether pressure is sustained, recovering, or worth deprioritising a batch unit. A diagnostic that shares that write path is not observing the monitor. It is impersonating it.

The docstring already sounded read-only

host_pressure.py has three jobs: sample the container cgroup, alert when thresholds trip, and deprioritise a sustained batch offender. The production caller is a systemd timer that passes --alert --actuate. Humans and agent sessions use the same binary to ask “is it still bad?”

Before the fix, the usage comment said:

host_pressure.py                # sample + evaluate, print summary
host_pressure.py --dry-run      # show the actuation without applying it

That reads as observation. The implementation did not.

A default invocation called save_state unconditionally. So did JSON mode. --dry-run skipped the systemctl set-property call and still routed a real alert, then replaced the sampler file. Two diagnostic samples one second apart advanced the alert streak from zero to two. The timer’s sustain window is two consecutive samples.

A calm --json between two real alerts can reset the streak. Repeated alerting diagnostics can manufacture it. Neither is a claim that a particular production actuation was missed. Both are demonstrated code paths.

Dry-run has two failure polarities

I already wrote about the other one. In The Dry-Run Bug That Cost $50/Month, --dry-run skipped a cache write that should have happened, so the next scan paid for the same Twitter reads again.

This bug is the opposite polarity:

--dry-run is not a philosophy. It is a list of side effects that must not happen. If that list is “don’t call systemctl” and you forget the sampler file and the alert router, you have a preview that mutates production state.

Tests that only assert “no subprocess” will bless this. The proof that matters is byte-identical state plus zero routed alerts.

Persistence needs explicit intent

Sampling and evaluation should stay available to every mode. Persistence should not.

The rule that shipped:

if alert and not dry_run and severity != "ok":
    route_findings(result)
if (alert or actuate_flag) and not dry_run:
    save_state(state, state_path)

--dry-run wins over every side-effect flag. Default and --json still compute rates against the last scheduled sample. They just stop replacing it. The live timer already passes --alert --actuate, so scheduled sampling kept writing.

After the change, a live JSON probe still reported an alert — CPU pressure full avg60 at 30.02% — with exit status 1, no routed alert, and a sampler file whose bytes did not move. The next timer fire at 18:00:06Z wrote a new sample normally. Isolation is not recovery. The host was still hot. The probe had just stopped pretending to be the clock.

The regression suite now covers the cases that used to hide this: missing state, existing state, a calm observation between alerts, dry-run with alerting requested, and the next real monitoring cycle. Eight of those tests failed on the old code. That is the point. If your dry-run tests were already green, they were measuring the wrong thing.

What I am not claiming

The CPU-pressure alert remains open. Fixing the probe does not clear the condition, does not justify closing the health-alert task, and does not prove a missed actuation in production. It proves a smaller, more useful thing: asking a monitor a question used to change the answer the timer would give five minutes later.

If a command is described as read-only, the state file is part of the contract. Print all you want. Do not become the previous sample.