The History Write Was Not the Gate

A voice call ends. The completion handler runs. The first thing it does is append a line to the history file — a JSON row recording the caller, the duration, the outcome.

That append fails. Full disk. Read-only mount. A flushing error at the wrong moment.

The exception propagates. Post-call scheduling does not run. The archive finalization step is skipped. The transcript is not promoted. Everything after the history write silently disappears.

The call completed. The record did not.

_on_call_end_locked was the cleanup handler. Its job was to finalize a completed call: persist the recent-call entry, schedule any follow-up, archive audio, promote the transcript to the long-term store.

It called _append_history_line first. Unconditionally. Any I/O failure in that function escaped as an unhandled exception and aborted the rest of the function body.

The history file is a tail-indexed JSONL log used for observability — last five calls injected into the next session’s context. It is not part of the call’s completion contract. A call that was answered, processed, and concluded had completed whether or not we wrote it down.

But the write was positioned before the work, so a logging failure became a finalization failure.

The fix is two lines:

try:
    self._append_history_line(...)
except Exception:
    logger.warning("history write failed; call finalization will proceed")

History is a side-write. Side-writes must not gate the main path.

The second bug hid behind the first

_append_history_line itself had a separate problem. It only bounded the row size when the row included an inlined context snapshot. Any other field could exceed _MAX_PAYLOAD_BYTES.

One of those other fields was caller on inbound calls. caller comes from customParameters.remote_party in the Media Stream event — a field from an unauthenticated Twilio connection, meaning attacker-length.

A row larger than _HISTORY_TAIL_BYTES is unrecoverable. The reader’s sliding window cannot reach the previous complete line. The writer cannot distinguish a torn tail from a valid oversized row. An oversized row silently blanks the entire index on the next read.

The fix is _bounded_history_note: clamp scalar fields to a safe length, drop the context snapshot if it would push the row over the cap, fall back to identity fields only if necessary. The cap becomes a guarantee for every caller, not just the ones that happen to include context.

Observability must not be in the critical path

The durability contract for a completed call is: archive, transcript, scheduling. That is the work that matters.

History is monitoring. Monitoring that fails should alert, not abort. If the history log fills up or the filesystem goes read-only, you want a warning in the logs — not a silent disappearance of every call’s post-processing for the duration of the incident.

This is the inverse of the durability problem. The ACK is part of the durability contract — an acknowledgement that claims durability must actually guarantee it. This bug was the opposite: a side-write that claimed no durability guarantee was acting as a gate.

Position observability writes after the critical path. Catch their exceptions. Log and continue. If you must write before the work, the write has to be load-bearing — and then it is no longer a side-write.