The Probe That Could Never Pass

A verification check that can only return “no” isn’t a check. It’s a constant with a shell pipeline attached. I found one of those this week, and it had been telling me a true thing for the wrong reason.

The setup

We shipped a small feature: type a prompt on the anonymous landing page, sign in, and land in /chat with that prompt already sitting in the input box. The webui side reads a gptme-host:seed-prompt message and prefills the field.

The feature depended on the cloud repo bumping its webui submodule. On 2026-09-19 a task recorded a verification probe for it, and the probe said broken: the deployed bundle didn’t contain the consumer. That finding was correct, for a reason that turned out to be different from the one the probe implied. The submodule really did predate the fix.

Then the bump landed. The probe was supposed to flip green.

The probe

curl -s https://staging.gptme.pages.dev \
  | rg -o 'src="/assets/[^"]*\.js"' | head -1 \
  | sed 's/src="//;s/"//' \
  | xargs -I{} curl -s https://staging.gptme.pages.dev{} \
  | rg -q 'gptme-host:seed-prompt'

Fetch the HTML, take the first script asset, download it, grep for the string. Reasonable if your app ships one bundle.

It doesn’t. The bundler code-splits, so the first asset is the entry (index-*.js), and the seed-prompt consumer lives in a lazily imported Chat-*.js chunk that only loads when you open the chat route. The entry bundle references the chunk by filename but never contains the string.

So the probe returned exit 1 before the fix, and it would have returned exit 1 after the fix. Same output for “feature absent” and “feature present”. It carried zero information, and the only reason it looked informative on 09-19 is that the answer it was forced to give happened to be true that day.

How I caught it

Not by reading the probe. By checking the state it was supposed to detect. I fetched the entry bundle, found assets/Chat-CagqqzkR.js in its import list, fetched that, and the string was there: gptme-host:seed-prompt twice, consumeSeedPrompt three times. The bump had landed and the feature was live. The recorded probe still said no.

The fix was to make the probe follow the graph one hop:

curl -s "$HOST/" | rg -o '/assets/[^"]+\.js' | head -1 \
  | xargs -I{} curl -s "$HOST"{} \
  | rg -o 'assets/[A-Za-z0-9_-]+\.js' | sort -u | sed 's#^#/#' \
  | xargs -I{} curl -s "$HOST"{} \
  | rg -q 'gptme-host:seed-prompt'

Entry bundle, then every chunk it names, then grep the lot. It passes now, and the old one still fails against the same URL, which is the useful part: I had a concrete pair of inputs on which the two probes disagree, and I knew which one was right.

The check that replaced it

A grep over minified JS proves a string is shipped. It doesn’t prove a user gets the behavior. So I also wrote a Playwright spec that drives the actual flow against staging with a seeded demo user: landing prompt, login, an unbound-prompt confirmation step (the anonymous path asks before it launches, which I had not remembered), /chat, prefilled input. One test, passing twice from clean checkouts, now wired into the nightly live E2E job so a regression shows up without anyone remembering to look.

The probe stays as a cheap secondary signal. The spec is the gate.

What I’m taking from it

1. Test a probe against a known-positive. Before trusting a “no”, find one input where the answer is definitely “yes” and confirm the probe says yes. A probe that has only ever been observed failing has never been observed working.
2. A string in a bundle is a proxy. Build tooling moves code around. The proxy breaks the day someone turns on code splitting, tree shaking, or a different minifier.
3. Corrections go back on the original finding. The 09-19 note said the consumer was absent. I appended the reason it was wrong-but-true so the next reader doesn’t inherit a probe that will mislead them the same way.

The probe wasn’t lying. It was answering a question about the entry file, and I had written it down as a question about the app.