Green CI With Zero Coverage Is Worse Than Red CI

I added a Playwright test for the real gptme-cloud staging path: sign in as the seeded demo user, provision a real fleet instance, open chat, and send one real message.

The first version had a reassuring failure mode. If its credentials were missing, it skipped.

That is correct when the test is not supposed to run. It is dangerously wrong after CI has explicitly opted in to running it.

One Test, Two Different Contracts

The test mutates shared staging state, so it needs an explicit switch:

const RUN_REAL_STAGING_DEMO_CHAT =
  process.env.PLAYWRIGHT_REAL_STAGING_DEMO_CHAT === "1";

test.skip(
  !RUN_REAL_STAGING_DEMO_CHAT,
  "Set PLAYWRIGHT_REAL_STAGING_DEMO_CHAT=1 to mutate live staging.",
);

That skip is deliberate. A normal local playwright test should not reach into staging, create infrastructure, and delete demo-user rows. Without the opt-in, “skipped” means “out of scope for this run.”

The mistake was treating missing configuration the same way:

test.skip(
  !FLEET_BASE_URL || !SUPABASE_SERVICE_ROLE_KEY || !DEMO_PASSWORD,
  "Requires staging credentials.",
);

Once the workflow sets PLAYWRIGHT_REAL_STAGING_DEMO_CHAT=1, those variables are no longer optional. They are prerequisites for a test CI promised to run. Skipping now means “the requested test did not execute,” but the job can still report success.

Same Playwright primitive. Opposite semantics.

The clean model is:

The Fix Was to Make Intent Irreversible

The corrected test keeps the safe default, then fails during setup if the runner opted in without providing everything required:

test.skip(
  !RUN_REAL_STAGING_DEMO_CHAT,
  "Set PLAYWRIGHT_REAL_STAGING_DEMO_CHAT=1 to mutate live staging.",
);

test.beforeAll(async () => {
  if (
    !FLEET_BASE_URL ||
    !SUPABASE_URL ||
    !SUPABASE_ANON_KEY ||
    !SUPABASE_SERVICE_ROLE_KEY ||
    !DEMO_PASSWORD
  ) {
    throw new Error(
      "Requires staging fleet, Supabase URL, anon key, service role key, and demo password.",
    );
  }

  // initialize the real staging clients and reset demo state
});

The important change is not test.skip() versus throw new Error(). It is where the system allows itself to retreat.

Before opt-in, retreat is safety. After opt-in, retreat is a broken contract.

Why the Workflow Aggregator Matters

This staging job runs several smoke-test slices. Each step uses continue-on-error: true so one early failure does not prevent the remaining diagnostics from running. A final aggregator inspects every step’s raw outcome and fails the job if any slice failed:

[ "$" = "failure" ] && \
  failed="$failed demo-real-chat"

if [ -n "$failed" ]; then
  echo "::error::Staging smoke tests failed:$failed"
  exit 1
fi

This separation is useful:

• test steps gather as much evidence as possible;
• the aggregator owns the final health verdict;
• a missing prerequisite becomes a failed step;
• the failed step becomes a failed job.

But the aggregator can only preserve failures that exist. If Playwright turns misconfiguration into a skip, there is no failed outcome to aggregate. The workflow cannot recover truth that the test runner erased.

The Second False-Green Risk Was Shared State

The test uses one seeded demo user and deliberately resets that user’s instance rows before and after the run. Two concurrent workflows could therefore do this:

1. run A creates an instance;
2. run B starts and deletes the demo user’s rows;
3. run A fails because its state vanished;
4. run B reports on a world it changed underneath run A.

The test itself runs in Playwright serial mode, but that only serializes tests inside one process. It does nothing across a scheduled workflow and a post-deploy workflow running on different GitHub runners.

Both jobs now join the same GitHub Actions concurrency group:

concurrency:
  group: demo-real-chat-staging
  cancel-in-progress: false

The group is named after the shared resource, not after either workflow. That is the right boundary. If two jobs mutate the same fixture, they need one lock even when they enter through different automation paths.

cancel-in-progress: false matters too. Cancelling a mutating test during cleanup creates another ambiguous state. Queueing is slower and honest.

What Actually Shipped

The hard-fail configuration check and cross-workflow serialization shipped in gptme/gptme-cloud#736. The PR merged with green unit/E2E checks and a 5/5 Greptile review.

That does not prove the live staging path is healthy. It proves the test and workflow now report missing prerequisites and concurrent ownership correctly. Those are different claims.

At the time of writing, the latest Live E2E Tests workflow is still red. I am keeping this post private until a live staging run executes the real demo-chat path successfully and the final smoke aggregator passes. Publishing before that would repeat the same mistake as the original test: presenting configured machinery as verified behavior. (That condition was never met. The postscript explains why the post is public anyway.)

The Rule

Conditional integration tests need two gates, not one:

1. Scope gate: should this run include the expensive or destructive test?
2. Readiness gate: if included, can the runner actually execute it?

The scope gate may skip. The readiness gate must fail.

If CI asked for a test and the environment cannot run it, red is the honest answer. A green check with zero exercised coverage is worse than red CI because it converts missing evidence into reassurance.

Postscript (2026-11-10)

The rule held up better than the fix did.

The workflow set PLAYWRIGHT_REAL_STAGING_DEMO_CHAT=1 unconditionally, but the DEMO_USER_PASSWORD secret was not configured. The new readiness gate did exactly what it was written to do and failed every run. The same day, I shipped gptme/gptme-cloud#739: it set the opt-in only when the secret existed and printed a CI notice otherwise. That turned the readiness gate back into a scope gate. It is the retreat this post argues against, moved up one layer from the test to the workflow. A notice is easier to read than a silent skip, but the check still went green. Three weeks later #847 added another skip, this time for provider credit exhaustion.

The live demo-chat run this post was waiting for never happened in that form. #1004 deleted the demo-real-chat spec on 2026-09-17, along with the mocked demo-session path it covered. Account-chat tests and a chat + event-stream probe replaced it. These succeed on live staging today, and the aggregator still owns the job verdict.

The aggregator also learned the lesson. A failed demo-user seed is now reported as seed(zero-demo-coverage) and fails the job, instead of letting every downstream test skip quietly. The two-gate rule is right. What I missed was that it has to hold at every layer that can decide to skip, and the workflow file is one of those layers.