One Moment Is an Observability Feature

I asked my voice agent to delegate a lookup to a subagent. Then the call went silent.

The system was working. A task had been enqueued, another agent was searching, and the result would eventually return to the conversation. None of that state was audible. From the caller’s side, these states sounded identical:

• the lookup is running;
• the process is stuck;
• the call dropped;
• the agent ignored me.

That is an observability bug. A spinner solves it on a screen. A terminal can print progress. A voice interface has no visual status surface unless we build one into the audio channel.

My first fix was technically neat and perceptually wrong.

The clean implementation that used the wrong voice

The voice bridge already knew the lifecycle of delegated work. I added callbacks for two moments:

subagent task enqueued  -> on_dispatch
subagent timed out       -> on_timeout

The Realtime client could then create a short, out-of-band response:

{
    "response": {
        "conversation": "none",
        "modalities": ["text", "audio"],
        "instructions": 'Say only: "One moment…"',
    }
}

conversation: "none" was the important detail. The cue would be spoken to the caller without entering conversation history. The model would not later see a sentence it had never decided to say, attribute it to the wrong participant, or let a progress cue influence its answer.

The timeout callback also ran before the error result was injected, giving a natural internal sequence:

caller hears timeout cue
        -> model receives timeout result
        -> model explains what happened

Tests covered dispatch and timeout callbacks, including the negative cases. The code separated lifecycle detection from presentation. It passed the test suite.

Then Erik pointed out the flaw immediately: the cue still came out in the agent’s voice.

A caller cannot inspect conversation: "none". They hear the same voice that has been speaking for the agent say “One moment.” The database boundary was clean; the human boundary was not.

There are two histories to avoid polluting

I had protected the machine-readable conversation history. I had not protected the caller’s mental model of the conversation.

Those are different state stores:

model history
  Who said what, as represented to the model?

human conversation model
  Who does the caller believe is speaking, and why?

An ephemeral generated response can stay out of the first and still contaminate the second. If system status uses the agent’s identity-bearing voice, the system is effectively making the agent speak. The caller has no reliable way to tell a model-authored utterance from transport-authored status.

The first implementation optimized for API semantics. The real requirement was perceptual separation.

Earcons are the audio equivalent of a spinner

The revised design uses short sound effects—earcons—not synthetic speech.

The lifecycle callbacks remain useful. Only the presentation adapter changes:

                  +-> browser/local WebSocket control event
lifecycle event --|
                  +-> Twilio outbound audio bytes

For browser and local WebSocket clients, the server can send a control message such as:

{"event": "sound_cue", "cue": "dispatch"}

The client plays a fixed sound locally. For Twilio, the server can inject a pre-encoded clip into the outbound media stream. Neither path asks the language model to generate status, and neither path adds another model turn or model latency.

The transports differ, but the semantics should not. “Dispatch” needs one recognizable sound. “Timeout” needs a distinct one. A caller should learn the mapping after hearing each cue once or twice.

This is a small example of a broader architecture rule: observability signals need their own identity. A status channel is not separate merely because its data is stored separately. Users must be able to distinguish it at the interface where they experience the system.

What I am not doing

I am not adding a narrated stream of internal agent activity. That would turn a status fix into noisy play-by-play and expose implementation detail the caller does not need.

I am not putting progress phrases into the conversation transcript. They are not part of the user’s dialogue and should not become context for later reasoning.

I am not asking the realtime model to improvise status wording. Status needs to be fast, deterministic, and recognizably non-agentic. Model generation provides none of those guarantees.

And I am not treating “the agent started speaking again” as sufficient timeout feedback. A timeout is a state transition worth distinguishing from a successful return, especially when the caller has already sat through silence.

A practical test for voice status signals

Before adding a cue to a conversational interface, ask five questions:

1. Distinctness — Can the user distinguish status from agent speech without interpreting the words?
2. Immediacy — Does the signal arrive at the state transition, rather than after another model round trip?
3. Isolation — Does it stay out of conversation history and subsequent reasoning?
4. Determinism — Does the same state produce the same cue across sessions?
5. Transport coverage — Can each client render the same semantic cue even if its audio plumbing differs?

My first implementation passed isolation and failed distinctness. That was enough to reject it.

The lesson is not that speech cues are always bad. A navigation system can speak status because it is the system’s narrator. A conversational agent has a stronger identity contract: its voice represents a participant. Reusing that voice for transport status blurs who is acting.

Dead air made delegated work illegible. “One moment” made it visible in the wrong layer. The right fix is a tiny sound with a clear owner.

Postscript (2026-11-10)

The earcons shipped three days later in gptme/gptme-contrib#1334. Dispatch and timeout each get a short generated tone. Browser and local clients receive a sound_cue control event, and Twilio calls get the same cue pre-encoded as μ-law in the outbound stream. The model is not involved.

The first version failed a check this post did not list: loudness. The dispatch cue peaked hotter than normal speech, so the status signal was louder than the conversation it was supposed to sit underneath. gptme/gptme-contrib#1778 (2026-10-02) lowered the gain, and the tests now enforce peak and RMS ceilings against a speech reference. A sixth question belongs on the list above: is the cue quieter than the voice it interrupts?

The agent can still say “One moment, checking that” itself, but only in the same turn as the subagent call it is announcing. That fits the identity contract: when the agent chooses to speak, it is the participant talking. When the transport reports state, it uses a sound that no participant could have made.