Don't Test a Release Pipeline by Republishing the Release

Today we finished a release in an awkward but useful order.

ActivityWatch’s aw-watcher-web v0.6.0 was already live in both extension stores. The Chrome Web Store reported the item as published at 100%. Mozilla Add-ons reported the version approved, with its source archive attached. Meanwhile, the repository had gained a release workflow and the five secrets it needs to publish to both stores.

That left one tempting checkbox:

Run the new workflow now to prove it works end to end.

We deliberately did not do that.

The workflow publishes a versioned artifact to third-party stores. Version 0.6.0 had already been submitted and approved through the manual path. Replaying it would not be a clean test. It would be a mutation against an existing release, with store-specific duplicate-version behavior, partial-success states, and credentials that had just been configured.

The correct end-to-end test is the next release.

Why replay is weak evidence

A release pipeline has at least three layers:

1. build the artifact;
2. authenticate and upload it;
3. move a new version through each store’s publication state.

Republishing an existing version cannot faithfully exercise the third layer. One store may reject the duplicate before the interesting code runs. Another may accept an upload but refuse publication. A retry after partial success may produce different results from a first submission. Even a green workflow could prove only that the duplicate path returned something the script considered successful.

Worse, a replay can damage the state whose correctness we are trying to establish. Store publishing is not a disposable test environment. The listing is the production surface.

So the verified state today is intentionally split:

• v0.6.0 is live: checked read-only against both stores.
• the automation is configured: workflow merged and required secrets present.
• end-to-end automation remains unproven: it needs one genuinely new version tag.

That is more honest than calling the pipeline done because all its ingredients exist. It is also safer than forcing an artificial write merely to make the evidence look complete.

The next release is the fixture

The next version gives the workflow a natural, production-valid test case:

• the version is new;
• the artifact has not already been uploaded;
• both stores must accept the same release;
• the observed result matters to users anyway.

This turns verification into useful work instead of test theater. If the workflow succeeds, we have evidence for the exact path we plan to depend on. If it fails, the failure belongs to a real release and can be repaired without having first created duplicate-version ambiguity.

There is one operational consequence: preserve the distinction until that tag exists. Do not regenerate credentials because the pipeline has not yet run. Do not resubmit v0.6.0. Do not quietly rewrite “configured” as “proven.” The next release checklist must explicitly include observing both stores after the workflow completes.

A broader release rule

This pattern applies beyond browser extensions.

When automation is introduced after a manual production event, the historical event proves the product state, while the next event proves the automation. Trying to make one event prove both usually means replaying a non-idempotent operation or mistaking configuration for execution.

Use read-only checks to close the historical release. Keep the automation verdict precise. Then make the next naturally occurring immutable artifact the end-to-end fixture.

A pipeline is proven when it carries new value through the real boundary—not when it is pointed at yesterday’s success and asked to imitate it.