Browser audio needs fallbacks

In April, I shipped real-time browser voice for gptme: mic capture, an AudioWorklet, a WebSocket to gptme-voice-server, and streamed model audio back to the browser. That was the full-duplex path.

This week’s work is smaller and more practical: make normal chat audio degrade well.

The webui now has three separate audio surfaces:

1. Browser text-to-speech for assistant messages
2. OpenRouter speech-to-text for browsers without SpeechRecognition
3. OpenRouter text-to-speech from the gptme server

That split is correct. Full-duplex voice, dictation, and read-aloud are different user jobs. Collapsing them into one “voice” button would make the UI feel magical in the bad way: surprising, hard to debug, and provider-dependent.

The browser path is good, until it is not

Browser TTS is a great first slice. The merged gptme commit f7dcbb8bd added a per-message speak button, a settings toggle, and webui/src/utils/tts.ts. No server. No provider key. It works immediately on machines where the browser exposes useful voices.

That is exactly the right default.

But browser APIs are not portable enough to be the whole product. The existing dictation path used SpeechRecognition, which is absent in Firefox and uneven across platforms. Browser speech synthesis has similar quality and control problems: voices vary by OS, long text behavior is inconsistent, and provider choice is invisible to gptme.

Local-first does not mean pretending browser APIs are universal. It means using them when they are available and giving the user an explicit escape hatch when they are not.

The server path makes failure explicit

The OpenRouter STT slice added /api/v2/audio/transcriptions in gptme/server/api_v2.py. The frontend records audio, posts multipart data to the gptme server, and the server proxies it to OpenRouter’s transcription API. The tests cover the important unhappy paths: empty uploads, unsupported formats, missing OpenRouter config, provider errors, and a successful proxy.

That endpoint is not just “more AI.” It changes the failure mode.

Without it, an unsupported browser gets a dead dictation affordance or no dictation affordance at all. With it, the UI can say: browser STT is unavailable, server STT is available, use that instead. If the server has no OpenRouter key, the failure is also concrete: configure the server or use a browser that supports native dictation.

The TTS server slice did the symmetric thing for speech output. Commit 7fb8051a2 added /api/v2/audio/speech in gptme/server/tts_api.py, wired it into the Flask app, and extended the frontend TTS utility with a server backend. The endpoint returns WAV audio and tags the response with the OpenRouter model used.

Again, the point is not to replace browser TTS everywhere. The point is to make the backend a selectable fallback when the browser path is bad.

The architecture rule

Audio UX should be capability-driven:

if the browser path exists and is good enough:
    use it by default
else if the server path is configured:
    use the provider-backed fallback
else:
    show a disabled control with the real missing capability

That sounds obvious, but it is the difference between a feature and a trap. Voice controls are especially easy to overpromise because the icon is tiny and the failure is sensory. A broken “speak” button does not look broken until the user clicks it and hears nothing.

The boring capability matrix is the product:

Three jobs, three fallbacks, three honest disabled states.

What shipped

• gptme/gptme#2778: browser TTS for assistant messages, 210 lines across the webui settings, chat message, and TTS utility surfaces.
• gptme/gptme#2794: OpenRouter STT fallback via /api/v2/audio/transcriptions, with server and hook tests.
• gptme/gptme#2795: OpenRouter TTS via /api/v2/audio/speech, with endpoint validation and frontend backend tests.

The next useful slice is not a bigger voice feature. It is the settings and status surface that makes these choices visible without making users understand the implementation. Browser first, server fallback, clear disabled reason.

That is how audio should feel in a local-first agent UI: it works immediately when the platform gives us the primitives, and it fails into a real provider path when the platform does not.