The First Byte Did Not Mean Progress

My AI review calls sometimes sit until their hard deadline. A first-byte timeout looked like a useful way to recover that wasted time: if the provider hasn’t started answering, stop waiting earlier.

Then I measured where the waiting happened.

Across roughly 24 hours, every one of the 33 classified timeout attempts received its first byte within 8.3 seconds. Some successful calls took longer than that to receive theirs. A first-byte cutoff generous enough to preserve every observed success would have caught none of those 33 failures.

I kept the deadlines and changed the measurement.

The failures had already answered

The census began at the first-byte instrumentation’s deployment boundary. It contained 730 attempt records:

Four timeouts lacked the timing and phase fields. They remain unclassified. The other 33 all carried the same phase: stream.

That label sounded more informative than it was.

The client requested a non-streaming JSON response. Its reader waited for headers, read one body byte, then blocked reading the rest to EOF. Here, stream meant that the first byte had arrived and the remaining read had not finished. It did not establish that the model had generated useful text, or that an SSE token stream had begun.

Even a single JSON-prefix byte or whitespace would satisfy that test. I did not capture the stalled responses’ actual first-byte contents.

Two shortcuts the data rejected

The slowest first byte among the classified timeouts arrived after 8.291 seconds. The slowest among successful attempts arrived after 22.819 seconds.

A global first-byte limit that preserved all observed successes would allow every classified failure past it. A ten-second limit would cancel some observed successes and catch none of these 33 failures. More aggressive cutoffs could catch failures too, at the cost of successful calls. This window did not support a cutoff that both caught these failures and preserved every observed success.

The other tempting shortcut was a shorter total deadline. Successful calls complicated that too: the longest fanout review took 882 seconds, and the longest local review took 785 seconds. A substantially lower global cutoff would cancel work that currently completes.

We could decide those successful reviews are too slow to be worth keeping. That is a product decision about acceptable latency and lost reviews. It is a different justification from claiming that elapsed time identifies a stalled response.

These observations describe attempts in one window. Retries and multiple review passes are correlated; 730 rows are not 730 independent user sessions. The census also separated consumers and providers. An aggregate result does not justify changing every route’s policy.

Measure the silence between bytes

The next patch reads the same response incrementally. It records returned body bytes, read events, the largest observed gap, the age of the last byte, and whether the transport completed. It retains the existing hard deadlines.

This gives us a way to distinguish a long response that continues delivering bytes from one that stops delivering them. It still does not measure useful model output: whitespace counts, and network buffering affects when the reader observes progress.

Timeouts need another distinction. If a request has been silent for 120 seconds when we stop observing it, we know the silence lasted at least 120 seconds. We do not know when the next byte would have arrived. The new observations mark those unfinished intervals explicitly rather than treating them as completed gaps.

Local HTTP fixtures verified successful, silent, and truncated responses. A deterministic slow-progress fixture ran through 800 seconds of virtual time without an idle abort. Those tests verify the reader’s behavior. They cannot tell us how long a real provider may pause before completing a valid review.

The next decision therefore waits for natural observations: at least 100 successful attempts and 10 instrumented timeouts, with long successful reviews represented and enough evidence in each consumer/provider partition under consideration. That floor starts the analysis; it does not guarantee a safe threshold.

The body-progress instrumentation has landed. The natural body-gap readout is still pending, and no idle-abort threshold has been chosen. I have no measured wall-clock savings to report yet.

The first-byte data was useful because it ruled out a plausible intervention. The new data has a sharper question to answer: does observable silence separate failed calls from successful ones well enough to justify cancellation?

Postscript (2026-11-10)

The natural readout arrived four days later, well past its floor: 1,509 fully instrumented successes and 139 instrumented timeouts. It answered the question with a no.

The longest gap inside any successful response was 4.07 seconds. The longest terminal silence in any timeout was 2.54 seconds. Every timeout was still receiving bytes when its hard deadline cut it off, with a median of 200 body-read events per attempt. A threshold above the longest successful gap would have caught none of the 139. Anything below the longest timeout silence would have cut off 1,439 of the 1,509 successes. The ordering held in every consumer that had both outcomes and across the providers.

So the timeouts were long responses that were still producing bytes, not stalled ones, and silence could not tell them apart. We added no idle-abort threshold and kept the hard deadlines. As this post said would happen, the first-byte data ruled out one intervention and the body-gap data ruled out the next.