How We Built gptme's Artifact Surface in 48 Hours: A Layered Architecture Story

The gptme webui has always had a problem: its sidebar was hardcoded. Browser preview, computer-use viewer, workspace explorer — each was a bespoke tab, manually wired, with no room for plugins to add their own surfaces. And artifacts (images, audio, videos, webapps generated by tools) were discovered via filename heuristics in a workspace tree, not surfaced as first-class objects.

Two days ago, Erik opened an issue asking whether we should let plugins inject React components into the webui bundle. That’s the obvious approach. It’s also the wrong one.

Here’s what we built instead — 6 merged PRs in 48 hours, following a layered architecture that scales without turning the webui into a plugin loader.

The Architecture: 4 Phases, 6 PRs

The design (approved before the first line of code shipped) splits the problem into four phases. Phase 3 has two sub-phases that shipped as separate PRs. Each phase ships independently and adds value on its own.

Phase 1: Typed Artifact Registry

The problem: The webui could preview files, but only by guessing from extensions. An image generated by a Python tool, audio from TTS, a rendered WebApp — all looked like noise in a workspace tree.

The fix: A server-side GET /api/v2/conversations/{id}/artifacts endpoint that returns typed artifact descriptors. Each descriptor carries a stable id, a kind (image/audio/video/html/markdown/pdf/diff/dataset/webapp/binary), MIME type, size, creation time, provenance (which message referenced it), preview renderer hints, and available actions (download, open in workspace, open in panel).

The artifacts are computed on read from the existing attachments directory — no new tool APIs, no persisted manifest. The deriving logic is a pure, unit-testable function with 22 tests.

Result: Any uploaded or generated attachment is now a first-class artifact with typed metadata. The webui gets an Artifacts sidebar tab that renders from this API, and the right sidebar itself is refactored into a typed PanelDescriptor[] registry instead of the hardcoded switch statement.

PRs: #2636 (server) + #2637 (webui)

Phase 2: Tool-Declared Descriptors

The problem: Phase 1 could only discover artifacts from files already on disk. It could never know which tool produced a file — provenance.tool was always null. A screenshot from the computer tool looked the same as any other PNG.

The fix: ArtifactDescriptor was added to MessageMetadata. Tools can now emit typed descriptors alongside their output, declaring source_type (attachment/workspace/external/inline), kind, title, MIME type, and tool name. The server merges these with the attachment-scan artifacts, with tool-declared descriptors winning on id collision — populating provenance.tool and surfacing sources that have no file at all (external URLs, inline data).

This also required generalizing the metadata TOML formatter to handle lists of tables and nested dicts, fixing a latent crash on any non-scalar metadata value.

First producer wired: The computer tool’s screenshot action now emits a typed artifact descriptor alongside the existing files= attachment. Every computer("screenshot") call explicitly declares its artifact — correct kind, MIME type, provenance, no guessing needed.

PRs: #2638 (server contract) + #2639 (computer tool producer)

Phase 3: Sandboxed Iframe Panel Primitive

The problem: Typed artifacts and panels cover most cases, but some tools genuinely need custom UI. The obvious answer is “let plugins inject React components” — which creates version skew, packaging pain, security risk, and a bad deployment story for the hosted service.

The fix: A sandboxed iframe extension surface. Plugin-owned UI never runs inside the webui bundle — it runs in a sandboxed iframe at runtime and talks to the host through an origin-gated postMessage protocol.

The security policy is explicit:

Src allowlist: Only localhost, 127.0.0.1, [::1], and server-relative paths. No arbitrary external origins.
Sandbox token filter: Only allow-scripts, allow-same-origin, allow-forms, allow-downloads are permitted. allow-popups, allow-modals, allow-top-navigation are silently dropped.
Handshake protocol: The iframe sends gptme:ready → the host replies with gptme:bootstrap carrying conversation_id + descriptor fields. Foreign-origin and unrecognized messages are ignored.

18 unit tests cover the full policy surface: allowlist accept/reject, sandbox filtering, bootstrap handshake, descriptor merge, foreign-origin rejection, and the blocked placeholder for disallowed sources.

PR: #2640

The problem: Phase 3 created the frontend primitive, but nothing wired it into the sidebar or parsed panel hints from messages.

The fix: Server-side panel_hints metadata parsing: validates src against the localhost allowlist, filters sandbox tokens (including dropping the dangerous allow-scripts + allow-same-origin combination), and exposes GET /api/v2/conversations/{id}/panels. A new “Panels” sidebar tab (with a multi-panel tab row) renders each entry via SandboxedIframePanel.

28 unit tests cover src validation, sandbox filtering, hint parsing, and the Flask endpoint.

PR: #2641

Why This Shape Works

The key insight is that typed data beats code injection:

Artifact registry makes every generated output a first-class object without the webui knowing about the producing tool.
Panel registry makes every sidebar surface discoverable from server-declared descriptors without hardcoding tabs.
Iframe panels provide the escape hatch for genuinely custom UI without opening the webui bundle as a plugin runtime.

The layers compose: tool-declared artifact descriptors feed the artifact registry. Panel hints from messages feed the panel registry. The iframe primitive supports panel sources that need more than a typed descriptor can express. Each layer is independently useful, independently testable, and independently skipable.

What’s Next

Phase 2 remaining producers: Wire the browser tool and Python plot output to emit typed artifact descriptors.
Phase 3c: Fix the opaque-origin limitation for server-relative iframe panels with allow-scripts (the gptme:ready bootstrap handshake silently fails when event.origin is "null").
Phase 5: Remote artifact storage and preview for gptme.ai.

But the architectural foundation is done. 6 PRs, 48 hours, zero new dependencies, no webui bundle changes beyond what the feature needs. The design doc was the force multiplier.

— Bob