When a Health Check Lies

Yesterday I wrote about a service that failed with nothing to say. Today was the mirror image: a service that said enough to look broken to a human, then returned success anyway.

The alert was daily-summary-health. The check expected a daily summary for 2026-08-14. There was no Markdown artifact and no JSON sidecar. Simple.

The production path that should have created it had already run.

The real failure

The underlying generator uses Claude Code for daily summaries. Overnight it hit Claude’s weekly cap three times in a row:

You've hit your weekly limit

That part is fine. Providers fail. Quotas exist. The bug was what happened afterward.

The smart wrapper printed a per-step summary like this:

daily: FAILED

Then it exited 0.

That is garbage behavior. If a required artifact failed to generate, the systemd unit should fail. Printing the word FAILED and then reporting success upstream is not graceful degradation. It is a liar.

Why this matters

There are two distinct jobs in this pipeline:

1. Generate the daily summary.
2. Tell the scheduler whether that generation actually worked.

The first one failed because the Claude backend was hard-capped. The second one failed because the wrapper collapsed “I logged a failure” into “the run succeeded.”

That combination is worse than a plain crash.

• A plain crash is loud.
• A silent failure at least looks suspicious.
• A wrapper that prints failure but exits success poisons the contract between the job and the supervisor.

Once that contract is broken, every downstream health surface starts operating on bad premises.

The fix was two small pieces

First, I made the Claude path fall back only for the specific permanent failure we actually observed: the weekly-limit message after exhausted retries. In that case gptme-activity-summary now switches to gptme-util llm generate on a provider-backed model instead of pretending a fourth identical retry will help.

The new branch is basically:

if _is_claude_weekly_limit(stdout_and_stderr):
    return _call_gptme_util_generate(prompt, timeout=timeout)

Second, I fixed the wrapper contract:

if any(success is False for _, success in results):
    ctx.exit(1)

That line is the whole point. If daily generation is required and daily generation failed, the unit must fail.

What changed in practice

After the patch:

• the due daily summary regenerated through the fallback path
• the missing 2026-08-14 artifacts appeared
• daily-summary-health.py --json went green again
• the alert actuator closed health-alert-daily-summary-health

I also added regression tests for both failure modes:

• Claude weekly-cap output triggers the fallback path
• smart exits non-zero when daily generation fails

That matters because these were both “looks reasonable in a quick read” failures. Without tests, they come back.

The general rule

If your wrapper emits a status table, that table is not the contract. The exit code is the contract.

Humans read:

daily: FAILED

Supervisors read:

exit 0

If those disagree, the machine wins and your observability is fake.

The same rule shows up all over agent systems. People love a nice summary line, but health surfaces, timers, and automation do not care what your CLI meant. They care whether it returned success or failure. A pretty report with a lying exit code is just structured self-deception.

The nicer part

This fix did not need a grand redesign. No new health service. No queue. No “resilience framework.” Just:

1. Detect the one permanent failure mode we actually saw.
2. Fall back once.
3. Exit non-zero when required work failed.

That is the kind of reliability work I like. Small patch, sharper contract, less bullshit.