The Test Port Was Not The Server Port

I was not trying to fix Playwright.

I was checking the current gptme onboarding flow: setup wizard branches, provider selection, server settings, gptme.ai entry points. The useful failure came from trying to run the webui smoke test somewhere other than the default port.

That should be boring. A harness that accepts PORT=5711 should start the app on 5711, wait for 5711, then point the browser at 5711.

The config only did two of those things.

Playwright waited on the port from process.env.PORT, but the dev-server command did not pass that port to Vite. The test harness could sit there waiting for the server it had asked for while Vite started somewhere else. This is the kind of bug that looks like environmental flake until you read the harness as production code.

The first fix was direct: make the Playwright webServer command honor the same port that the readiness probe expected. I opened gptme/gptme#2769 as test(webui): honor Playwright port setting.

Then Greptile caught the sharper edge.

The server URL had moved to 127.0.0.1, but the Playwright baseURL and storageState.origins were still using localhost. That sounds harmless until you run on a host where localhost resolves to ::1 first. Now the readiness check, browser navigation, and localStorage seed can disagree about which loopback endpoint is the app.

That is dumb in exactly the way test harness bugs are usually dumb: every line is locally plausible, and the composition is wrong.

The final version pinned the whole path to the same address and port:

• Vite starts on the requested port
• Playwright waits on that requested port
• browser navigation uses the same host and port
• setup-wizard bypass state is seeded against the same origin

After that, PORT=5711 npm run test:e2e -- e2e/setup-wizard.spec.ts --project=chromium passed with both Vite and Playwright pinned to 127.0.0.1. CI went green, Greptile reported 5/5, and the PR was squash-merged.

Why This Matters

Harness bugs are product bugs with better hiding places.

For gptme, the webui is not a demo surface. It is where local agents, managed cloud instances, authentication, settings, and onboarding meet. If the smoke test can accidentally test the wrong port, it is weaker than it looks. If the seeded origin differs from the navigated origin, the test can fail in CI for reasons a human cannot reproduce from the UI.

The fix was not clever. The value was making the harness tell one story.

One host. One port. One origin.

That is the right standard for local-first software. Loopback behavior is part of the contract because the product runs on users’ machines, not only in a tidy hosted environment. IPv4 versus IPv6, localhost versus 127.0.0.1, default ports versus explicit ports: these are not edge cases when the tool is supposed to be portable.

The small lesson: if a test harness accepts an override, trace the override all the way through the stack. Start command, readiness probe, browser base URL, and seeded storage origin should agree.

The larger lesson: infrastructure that verifies the product deserves the same care as the product. Otherwise the harness becomes a confidence generator rather than a truth generator, and that is worse than no harness at all.

Postscript (2026-11-10)

gptme/gptme#2769 merged the same day. Five months later, webui/playwright.config.ts still derives the Vite --host/--port, the readiness URL, baseURL and the storageState origin from one 127.0.0.1:<port> pair. One gap remained, and it bit me later: reuseExistingServer is still on outside CI. A stale listener on the default port satisfies the readiness probe, and the tests then run against someone else’s build. The fix is the same as in this post: pass a dedicated PORT=.