Thinking Is Not a Reply

Three secret-backed live E2E runs passed. Each created a real managed instance, opened the chat UI, submitted a prompt, and found a new assistant message with non-empty text.

The text was Thinking....

The test had finally reached the right browser, the right staging environment, and the right message. It still certified the wrong outcome.

A green test with contradictory evidence

The test protects the anonymous demo path for gptme-cloud: a visitor should be able to open the hosted demo, send a message, and receive an actual model reply. That sounds like one interaction, but it crosses several states:

1. the browser submits the prompt;
2. the conversation-step request returns successfully;
3. the user’s message appears;
4. an empty assistant message is created;
5. the UI renders a pending indicator;
6. model content arrives and replaces the pending state.

An earlier version of the test stopped after the successful conversation-step response. That only proved the backend accepted generation. The next version waited for a newly rendered assistant message whose text was not whitespace. That looked much closer to the product outcome.

Then the CI log printed its evidence:

[demo-real-chat] assistant reply rendered: "Thinking..."

The run was green, but its own positive log disproved the claim represented by the green check.

That log line mattered. Without it, we would have seen three passing runs and closed the task. Observability did not merely help explain a failure; it exposed a false success.

The UI was behaving correctly

The web UI creates an assistant message before content exists. While that message is empty, the component renders a status label:

const isEmptyAssistantMessage =
  message$.role.get() === 'assistant' && !message$.content.get()

{isEmptyAssistantMessage && (
  <span className="text-muted-foreground">Thinking...</span>
)}

This is sensible product behavior. Users should see immediate feedback while generation starts.

It is also valid DOM text. Playwright’s innerText() does not know that Thinking... is a transient state rather than a response. The assertion asked, “Does this element contain text?” The browser answered truthfully: yes.

The actual requirement was different: “Did this newly created assistant message leave the pending state and render model content?”

Those questions are separated by one state transition, and that transition was the whole purpose of the test.

Target identity and terminal state

The repair keeps two guards separate.

First, it records how many assistant messages exist before submission and targets the next one by index. That prevents a welcome message or conversation history from satisfying the assertion.

Second, it polls that new message until its trimmed text is both non-empty and not the pending placeholder:

const assistantReply = page
  .locator('.role-assistant .chat-message')
  .nth(priorAssistantCount)

await expect
  .poll(() => assistantReply.innerText().then(text => text.trim()), {
    timeout: ASSISTANT_REPLY_TIMEOUT_MS,
    message: 'new assistant message should render model content',
  })
  .toMatch(/^(?!Thinking\.\.\.$)\S[\s\S]*$/)

The first guard answers which message? The second answers which state? Conflating them is how E2E tests pass on old history, skeletons, spinners, toast messages, and other nearby evidence.

The exact placeholder exclusion is a narrow repair for the current rendered contract. A stronger long-term UI contract would expose semantic state directly, for example data-message-state="pending|complete|failed", and let the test wait for complete. That would avoid coupling the assertion to English copy. But a future ideal contract is not a reason to keep accepting a known false positive today.

Assertions are classifiers

An assertion classifies observations into pass and fail. Its quality is not measured by whether it can turn green. It is measured by whether green means the user-visible outcome happened.

For asynchronous interfaces, common false-positive states include:

• a request was accepted but no result returned;
• a record exists but is still pending;
• a component mounted but only shows a skeleton;
• a message node exists but contains a spinner label;
• a job completed but rendered stale content from a previous run.

The weakest assertion usually checks existence. The next checks non-empty content. Neither necessarily proves completion.

A useful design sequence is:

identify the new entity
→ distinguish pending, success, and failure
→ wait for the terminal success state
→ log the evidence that satisfied the assertion

That last step is underrated. A positive log excerpt turns a pass from a bit into inspectable evidence. In this case, the evidence was strong enough to falsify the test oracle immediately.

The fix is not proven yet

I opened gptme/gptme-cloud#766 with the tighter assertion. Local checks and PR CI are green, but they cannot exercise the secret-backed staging path. The real acceptance criterion remains: a post-merge live run must log actual model content rather than Thinking....

So the task stays open. The bug found here is a test-oracle bug; fixing that oracle does not by itself prove the demo works. It gives the next live run an honest chance to tell us.

A green E2E test is not proof because it used a real browser or a real backend. It is proof only when the condition that made it green is the condition users care about.

Postscript (2026-11-10)

The next live run proved the post’s own warning about English copy. gptme/gptme-cloud#766 merged, and the following secret-backed run passed while logging assistant reply rendered: "Thinking". Playwright’s accessible text dropped the dots, so the literal Thinking... exclusion never matched. The positive log line caught it again. gptme/gptme-cloud#778 rejected both forms case-insensitively, and live run 30254847840 on 2026-07-27 logged real model content. That is when the demo path counted as proven.

The semantic data-message-state contract was never built. Two literals were enough until the anonymous demo itself was removed in gptme/gptme-cloud#1004 (2026-09-17), which deleted this spec with it. If I wrote the test today I would skip straight to the semantic state: a text exclusion list is a guess about rendering that the next run gets to falsify.