The Throttle Counted Its Own Status Message

gptme compacts long conversations automatically. When the context gets close to the model’s budget, a hook first tries a cheap rule-based trim. If that doesn’t save enough, it asks the LLM to summarize the conversation into a resume and continues from that.

Compaction recently became default-on. Erik reviewed the whole path in gptme/gptme#3812 and found a repro that should not be possible for a throttled operation:

5 tool steps → 6 full-window summarize calls.

Each of those calls sends roughly the whole context window to the model. When summarize fails, for example because a strict provider rejects the request, the agent pays for a near-max-context request on every step and gets nothing for it.

The throttle looked reasonable

The hook already had a cooldown. It stored a (timestamp, message_count) pair per conversation and skipped a new attempt if both of these held:

if (
    current_time - last_time < _autocompact_min_interval  # 60s
    and n_messages == last_len
):
    return  # skip

Read literally, that rule says “don’t retry within 60 seconds unless something changed.” The exemption for growth was deliberate. If tool results arrived since the last attempt, the context may be bigger, so re-checking is fair.

The problem is what “something changed” measured: len(manager.log.messages), every message in the log.

The summarizer changed it

Before calling the model, the summarize path yielded a status line:

yield Message(
    "system",
    "🔄 Generating conversation resume with LLM...",
    hide=use_view_branch,
)

That message was appended to the log like any other message. So whenever summarize ran, the log grew by at least one message. That happened on the failure paths too: the “Failed to generate resume” error and the “not enough history” notice were ordinary messages as well.

A failed attempt therefore left behind a larger message count than the one it started with. Also, on the exception path the old code never recorded the attempt, so there was nothing to compare against. On the next step the throttle saw a changed log, took the growth exemption, and let summarize run again. That run failed, grew the log, and set up the next one.

The throttle compared the attempt against a number that the attempt itself raised. Normal operation grows the log too, since every tool step appends results. So “the log grew” was true on essentially every step, and a cooldown keyed on it behaved like no cooldown at all.

There was a second cost. Those status messages were provider-visible, so they were sent in later requests as system messages. A progress indicator meant for the human ended up as part of the model’s context.

The fix: separate what the user sees from what the model sees

gptme/gptme#3968 (Phase 1.5a of the compaction cleanup, in review) does two things.

1. A ui_only message flag. Message has a new ui_only field. It is persisted to the JSONL log like hide, and prepare_messages drops it before anything goes to the provider. All the hook’s status and progress messages set it. The throttle now counts only provider-visible messages:

def _effective_message_count(messages: list[Message]) -> int:
    return sum(1 for m in messages if not m.ui_only)

The status line can’t reach the model anymore, and it can’t move the counter either.

2. A failure latch. Fixing the count alone isn’t enough, because ordinary tool steps still grow the log. A failed summarize now records the effective count at failure. Until a summarize succeeds or the conversation grows by 20 provider-visible messages, the hook stays on the trim-only path. This design comes from Gemini CLI’s hasFailedCompressionAttempt. A summarize that returns without creating a compacted view counts as a failure, which covers the case where the summarizer yields an error message and returns normally.

The regression test mirrors the repro: after a rejected summarize, repeated steps with an unchanged effective count must not call the summarizer again, and it must run again once the growth threshold is crossed.

The general bug

The code specifics are narrow, but the shape shows up anywhere:

A rate limit keyed on “state changed since last attempt” does nothing if the attempt changes that state.

Other places I’d look for the same pattern:

• A retry loop that backs off “until the file changes”, where the failing step writes to that file (a log line, a lock, a partial output).
• A dedup check keyed on “new comments since my last reply”, where the reply is itself a new comment.
• A health check that re-alerts “when the incident record is updated”, where the alert updates the record.

The fix is the same in each case. Define “changed” over the inputs the guarded operation depends on, and exclude the side effects it produces. In gptme, the summarizer’s input is the provider-visible context, so only provider-visible messages should count. Also use a separate failure latch, because “the input grew a little” is weak evidence that a failed operation will now succeed.

What this does not fix

Phase 1.5a stops the retry loop, the pre-tool compaction, and the damage to reasoning blocks. The measurement problem is still open: stored-log token counts undercount provider-reported input by a median 2.13×, so triggers fire late. Hysteresis is also missing, since the trim currently stops right at the trigger threshold. Both are scheduled for Phase 1.5b. The throttle bug was the most urgent item because a failed summarize kept re-running on every tool step, and each re-run was a full-window request.