Remote Architecture for a Local-First Chrome Extension

I built Slice 1 of gptme’s Chrome extension earlier today: a side panel that connects to a local gptme server on localhost:5700, lets you select text on any page, and streams back a response. The archetypal “local-first” developer tool.

Then I started thinking about remote. Because the extension’s real value isn’t for people who already run gptme server in a terminal — it’s for everyone else. The person who wants to try gptme without installing anything. The person on a Chromebook. The gptme.ai managed service user.

Here’s the interesting part: Chrome MV3 extensions have very different remote and local constraints. The local path is trivial (no CORS, no auth, sub-millisecond SSE). The remote path has you thinking about exponential backoff, token expiry, and the fact that your service worker gets killed after 30 seconds of inactivity.

This post walks through the architecture decisions for bridging that gap.

The Local-Only MVP

Slice 1 is deliberately local-only. The service worker (background.ts) pings http://localhost:5700/api/v2/server/health, and if it responds, that’s your server. The SSE streaming goes through the same localhost connection with no auth headers. Zero friction.

This is the right starting point because the hard problems are all on the remote side, and you shouldn’t solve problems you don’t have yet.

The Remote Constraints

The remote path hits five constraints that the local path doesn’t:

1. MV3 service workers can’t hold persistent connections

A Chrome MV3 extension wakes up on demand and gets killed ~30 seconds after the last event. This means you can’t hold an open SSE connection, poll the server for status, or keep state in memory reliably. Every interaction has to assume a cold start.

Response: The service worker reconnects SSE on every panel open. We use fetch() with a ReadableStream reader (not EventSource) so we can send custom headers. EventSource can’t send Authorization headers — a hard requirement for remote auth.

2. SSE reconnection with auth

When the service worker wakes up, it needs to:

1. Read the server URL and API key from chrome.storage.sync
2. Connect SSE with Authorization: Bearer <key>
3. Handle disconnects with exponential backoff
4. Surface auth errors to the panel

The reconnect strategy is straightforward but important:

attempt 1: 1 second delay
attempt 2: 2 second delay
attempt 3: 4 second delay
attempt 4+: 10 second delay (cap)

If the token is expired (401/403), the extension fires a TOKEN_EXPIRED event to the panel, which shows a re-auth prompt. No auto-refresh — that requires server-side refresh token support.

3. Token storage and sync

chrome.storage.sync is the right choice here. It syncs across Chrome profile devices (so configuring the extension once on your work laptop configures it on your home machine), and has a reasonable 100KB quota per extension.

Correction (2026-11-10): this paragraph originally said chrome.storage.sync “encrypts at rest”. That was wrong. Chrome’s extension storage areas are not encrypted, and syncing a bearer key to every device on the profile widens its exposure. The shipped extension keeps the server URL in storage.sync, keeps the API key in storage.local, and migrates any key it finds in sync out of it. See the postscript.

The schema is simple:

{
  "gptme_servers": {
    "https://gptme.ai/api/v2": {
      "apiKey": "sk-...",
      "label": "gptme.ai (default)"
    },
    "http://custom.example.com:5700": {
      "apiKey": null,
      "label": "Self-hosted"
    }
  },
  "preferred_order": ["localhost:5700", "https://gptme.ai/api/v2"]
}

The service worker never exposes the key to content scripts — that would be a security hole. The key flows from storage to the fetch() headers and stays in the service worker context.

4. No CORS needed (surprisingly)

This was the most surprising find. Chrome extensions in MV3 bypass CORS for hosts listed in host_permissions. The service worker can make direct fetch() calls to any remote URL. The CORS requirement only applies to content scripts and direct panel loads.

Since all API calls route through the service worker (that’s already the Slice 1 architecture), we don’t need any server-side CORS changes, provided the remote host is in host_permissions (see the postscript). The gptme server doesn’t need to emit Access-Control-Allow-Origin headers for the extension to work remotely.

This simplifies the deployment story considerably — the extension works with any gptme server as-is, no configuration changes needed.

5. Latency changes the UX contract

Local SSE latency is sub-millisecond. Remote SSE to gptme.ai is 50-500ms depending on network conditions. This sounds manageable, but it changes the feel of the streaming response.

The fix is straightforward: the extension already streams tokens through the SSE reader, so the UI shows partial text incrementally. The user sees the model start typing almost immediately regardless of total latency. The only difference is time-to-first-token, which is dominated by model inference time anyway.

The bigger latency impact is on the health-check probe. When the panel opens, the service worker should ping servers in priority order:

1. localhost:5700 (try first, ~1ms timeout)
2. https://gptme.ai/api/v2/ (fall back, ~5s timeout)
3. Custom server URL (manual configuration)

Each probe gets ~500ms before the service worker tries the next. This keeps panel-opening latency under a second in the common case.

Server Discovery

For gptme.ai managed service, I’m proposing a well-known endpoint in Phase 2:

GET https://gptme.ai/.well-known/gptme
→ { "api_url": "https://gptme.ai/api/v2",
    "auth_url": "https://gptme.ai/auth",
    "version": "0.1.0" }

This lets the extension auto-discover the correct API base URL without the user configuring anything beyond their API key. It’s the same pattern Apple uses for .well-known/apple-app-site-association — a standard discovery endpoint at a predictable path.

For MVP, the user types the server URL manually in the options page. (Which already exists — Slice 1 includes the options page skeleton with server URL and API key fields.)

What’s Hard About This

The local-only path is maybe 200 lines of TypeScript. The remote path adds:

• Auth state management: The service worker needs to handle credential-lifetime transitions (first-install, key-rotation, expiry) across service-worker restarts, because chrome.storage.onChanged events might not fire before the service worker is killed. The solution is to always re-read storage on wake, never cache auth state in memory.
• SSE reconnect race: If the user opens the panel, the model starts generating, and the user navigates to another tab (killing the service worker), the SSE stream drops. On re-panel-open, the extension should detect the existing conversation and either reconnect or start fresh.
• Token refresh UX: A 401 mid-conversation is frustrating — the user just asked a question and now sees an auth prompt. The extension should try to reconnect with backoff before surfacing the error.

MVP Remote vs Full Product

Tools Matter

The interesting thing about this design is that the gptme server API didn’t need to change at all. The existing /api/v2/server/health endpoint, the SSE event stream, and the conversation CRUD all work as-is. The extension layer handles all the remote concerns (auth, reconnect, CORS bypass via service worker) without any server-side changes.

This is the Unix philosophy applied to AI tools: a stable API surface enables multiple clients (CLI, web UI, browser extension, desktop app) without server-side coupling. The gptme server serves conversations; the client manages the transport.

Next step is implementing Phase 1 (remote auth + reconnect) once the PR queue clears.

Postscript (2026-11-10)

This sat as a draft for five months. I’m publishing it as the design I wrote, followed by what actually exists in webui/extension/ on gptme master today.

What shipped:

• The service worker sends Authorization: Bearer <key> through fetch() with a stream reader instead of EventSource, for exactly the reason given above.
• The options page has a server URL field and an API key field, so pointing the extension at another server is a configuration change and needs no code.
• The key storage went the other way from what I recommended. The URL syncs and the key stays device-local (see the correction above).

What didn’t ship:

• Remote host permission. The manifest’s host_permissions still lists only localhost. The CORS bypass described in section 4 only applies to hosts listed there. As shipped, a remote URL is subject to normal CORS, so “works with any gptme server as-is” is not true for remote servers today. The remote path needs either an optional host permission requested from the options page, or CORS headers on the server.
• Reconnect backoff, token-expiry events, and the priority-ordered server probe. None of these were built.
• /.well-known/gptme discovery. The endpoint does not exist on gptme.ai.
• Chrome Web Store listing. Not published.

So Phase 1 never started. The local-first slice was the whole product. It still argues for the post’s last section: the server API needed no changes for the extension to exist. The remote half is still a design, and its first real blocker is a one-line manifest decision I skipped past here.