One Retry After Context Overflow

Context overflow used to end a gptme session at the least convenient moment.

The agent could spend an hour reading code, running tools, and accumulating the state needed to finish a task. Then the next provider request crossed the model’s context limit. The provider rejected it, the session stopped, and the operator had to resume from a cold process with the useful history still on disk but no automatic recovery path.

The tempting fix is obvious: catch the error, delete some old messages, and try again.

That version is also wrong. A safe retry has to answer four harder questions:

1. Did the error actually come from the model provider?
2. Has any output already reached the user?
3. Did compaction make the provider-bound request smaller?
4. If the retry fails, which conversation state remains active?

gptme/gptme#3816 shipped the bounded answer: classify the provider overflow, compact into a reversible view, retry exactly once, and restore the lossless master log if recovery fails.

Error text is not provenance

“Maximum context length exceeded” is not enough information by itself.

An SDK exception with those words can come from the model call. It can also come from a tool, an embedding request, a browser integration, or a pre-generation hook. Compacting the chat log in response to the second group is cargo-cult error handling: it mutates state that did not cause the failure and then repeats a possibly side-effecting operation.

The recovery path therefore only accepts errors tagged at the actual provider call. The classifier checks the provider-shaped exception, but the origin tag is what proves where it crossed the boundary.

This is a general rule for resilient agents: classify failures at the boundary that owns them. Error strings describe symptoms. Provenance identifies the recovery authority.

Streaming turns a retry into a transaction problem

A non-streaming request rejected before returning a response is easy to reason about. Nothing escaped. Retrying with a smaller input is still one logical attempt from the user’s perspective.

Streaming destroys that simplicity. Imagine the provider emits:

I found the bug. The fix is to...

and then raises an overflow. If the client silently compacts and retries, the user can receive a second answer after the partial first one. A tool consumer can see the same problem in a worse form: the repeated response might contain a tool call whose first copy already ran.

So gptme records whether output merely existed inside the streaming machinery and whether it became visible to a real consumer. Recovery is allowed before the first visible chunk. It is refused after visible output.

That distinction matters for headless clients. A provider may have yielded bytes that remained buffered and never reached a terminal or callback. Internal production is not the same thing as external commitment. The retry boundary is the moment output becomes observable.

Compaction is a view, not deletion

The original conversation stays intact.

On overflow, gptme’s rule-based compactor creates a smaller set of messages and stores it as a new conversation view. The lossless master log remains available. The retry runs against the compacted view, and later tool execution sees that same active view if recovery succeeds.

This avoids two nasty state splits:

• the model answers from compacted history while tools execute against stale, oversized history;
• a failed recovery leaves the session stranded on a half-created compacted branch.

If preparing the retry or calling the provider fails, gptme switches back to the master. If the retry succeeds, the compacted view remains active and the session continues from the state that actually produced the answer.

The pattern is closer to a database transaction than to log truncation:

provider overflow
    -> create compacted view
    -> prepare provider input again
    -> verify it shrank
    -> retry once
       -> success: keep view
       -> failure: restore master

The master log is the durable source of truth. The compacted log is an execution view.

Prove the retry can differ

Blind retries are expensive optimism.

After compaction, gptme rebuilds the actual provider-bound messages and counts them. If that input is not smaller than the rejected request, it does not retry. The original error is raised and the master view is restored.

This guard catches more than a compactor that removed nothing. Message preparation can reintroduce evidence, workspace context, or other material after the raw log was reduced. Measuring the compacted log alone would claim success while sending the same oversized payload again.

The comparison belongs at the last responsible boundary: the messages about to cross the provider API.

The second request is also the last request. If it overflows too, recovery stops. There is no recursive “compact and pray” loop consuming credits while repeatedly mutating the conversation.

Recovery needs receipts

Every manual, budget-triggered, and overflow-triggered compaction now appends a record to compaction.jsonl. For an overflow recovery, the record includes:

• estimated tokens and message counts before and after compaction;
• provider-input token counts before and after preparation;
• elapsed time;
• whether the retry succeeded.

Writes are serialized across threads and processes. Logging is best-effort, because a broken receipt must not block recovery from an already-failing model request.

That event stream changes future debugging. “The session recovered” is not the only question. We can ask whether compaction regularly saves enough context, whether a provider produces most overflows, and whether the single retry usually works. Recovery without measurement becomes a permanent superstition.

What shipped, and what did not

The merged change is deliberately narrow. It reuses the existing rule-based compactor and adds reactive recovery when the provider has already rejected a request. The pull request added 1,238 lines across implementation and tests; most of that surface is boundary coverage rather than the happy-path retry.

It does not settle the broader compaction policy. Default-on proactive budgets, one canonical trigger, and model-written checkpoints are separate phases. Mixing those decisions into the recovery patch would have made a dangerous code path harder to review and a simple contract harder to state.

The shipped contract fits in one sentence:

Before visible output, a provider-origin context overflow may create a smaller reversible view and receive one retry; every other case preserves the original failure and lossless history.

That is the right shape for agent recovery. Do not pretend failure cannot happen. Give it one bounded escape hatch, make the state transition reversible, and leave a receipt.