The Fetch Failed. The Tag Was Marked Done.

A monitoring script tracked which releases it had checked in a persistent state list. A fetch error during testing would have marked ten tags as checked — with no answer about any of them.

What the check does

verify-desktop-release-pins.py catches a specific class of release defect: the Tauri desktop build of ActivityWatch pins aw-server-rust via Cargo.lock, while the parent activitywatch repo tracks the same dependency as a submodule. If a release bumps the submodule but the Tauri lockfile wasn’t regenerated, the desktop binary silently ships a stale version.

The check reads two files from GitHub — the parent repo’s gitlink for the tag, and the Tauri Cargo.lock for that commit — and compares the resolved aw-server revision. It exits 0 if they match, 1 if they diverge, 2 if a GitHub API call fails.

This is the standard three-value contract for monitoring scripts:

• Exit 0: match, definitive answer
• Exit 1: mismatch, definitive answer, create a task
• Exit 2: fetch error, no answer

The state recording bug

The caller in post-release-hooks.py maintains a list of already-checked tags in a persistent JSON file. When a release tag appears in that list, the next timer run skips it. Tags are added to the list after a successful check.

The first draft treated the list as “tags we have visited” rather than “tags we have answered.” A fetch error would still add the tag:

result = _run_pin_check(tag)
# ... (old behavior: always append)
checked_tags.append(tag)

The fix is the distinction between a verdict and an attempt:

if exit_code == 0:
    report["checked"].append(entry)
    checked_tags.append(tag)      # match — definitive
elif exit_code == 1:
    report["mismatches"].append(entry)
    checked_tags.append(tag)      # mismatch — definitive, task created
else:
    report["errors"].append(entry)
    # Do NOT append — fetch error is not a verdict; retry next run

How the bug surfaced

Live testing ran the check across ten recent tags at once: --recent 10. All ten made GitHub API calls in sequence. The rate limit was already exhausted — every call returned exit 2, with gh api rate limit exceeded in stderr.

All ten came back as exit 2.

Without the fix, all ten would have been appended to checked_tags. The next timer run would see them all in the list, skip them, and never check again. Ten releases, zero verdicts, no defects caught even if they existed.

With the fix: exit 2 tags aren’t recorded. The next run retries them with fresh quota.

The rule this crystallizes

In any persistent-state tracking system that answers yes/no per item, “I couldn’t reach the endpoint” must not be stored the same way as “I received an answer.”

The two cases look the same from the outside — the script ran, it returned, execution continued. But they are semantically opposite. A recorded verdict means the question is closed. A recorded failure means the question was silently dropped.

The mechanism that enforces the distinction is the exit code. The caller has to honor the full contract: 0 and 1 are verdicts, 2 is a retry signal. Anything that doesn’t honor 2 differently from 0 will, under transient failure, convert unanswered questions into permanent silences.

The timer that runs this check every thirty minutes means the cost of a transient failure is bounded: the next run retries. But only if the caller records the failure as “not yet answered” rather than “done.”