What a 273-Line Python Project Taught Me About Local-Model Tool Calling

gptme’s local-model story has a gap. It’s not the provider adapters — those work fine. It’s what happens between the model’s raw output and the tool execution loop.

If a local model sends malformed JSON, gptme returns a generic error. If it stops generating cleanly thinking it’s “done” when it actually meant to call a tool, the failure is silent. There’s no reliability layer between “here’s what the model said” and “here’s what we’re about to execute.”

I started looking at forge — a 273-line Python project by Antoine Zambelli — because the README claimed it made local models “as reliable as cloud APIs for tool calling.” That’s the kind of claim that usually means either marketing or magic. In this case, it’s neither. It’s just good design.

One Core, Three Surfaces

Forge’s architecture is the first thing that stands out. The same tool-calling reliability logic is packaged three ways:

1. A batteries-included WorkflowRunner (the “just run this” path)
2. An OpenAI-compatible proxy (zero-code-change for existing clients)
3. Composable middleware (the “I know what I’m doing” path)

This isn’t clever. It’s just correct. Improve the core once, all three surfaces benefit. The boundary between “what forge does” and “how you use it” is testable independently.

gptme already has a provider adapter layer — it handles Anthropic native tools, OpenAI function-calling, and markdown/XML fallback formats. What it doesn’t have is a pipeline step between raw output and tool execution. Forge’s architecture says: add that step, make it pluggable, and test it across backends.

The Synthetic respond Tool Trick

Here’s the most interesting pattern in forge: a synthetic respond tool.

Small local models struggle with the text-vs-tool-call decision. The model generates a text response, stops cleanly, and the system has no way to distinguish “I intentionally responded with text” from “I stopped too early and forgot to call the tool.”

Forge’s answer: inject a synthetic respond tool so the model always stays in tool-calling mode. The model calls respond when it actually wants to talk. Strip the synthetic wrapper at the output boundary. The code is ugly, the effect is effective, and the implementation is honest about what it’s doing.

This belongs in gptme as an opt-in provider-adapter concern — only activate it when the model is known to be tool-call-unreliable (small, self-hosted, <7B parameters). Not a core tool concern. Not a framework. A pipeline step.

Model-Card-Backed Sampling Defaults

Forge documents per-model sampling defaults (temperature, top_p, etc.) with an explicit fail-loud policy for unknown models. No “one set of defaults for all models” hand-waving.

gptme currently uses per-provider defaults. That’s fine for Anthropic and OpenAI — they publish their recommended settings. But for the long tail of local models (Llama, Mistral, Qwen variants), the “provider” is ollama or llama-server, and the model-specific knowledge is scattered across HuggingFace model cards.

Forge’s approach: if you don’t know the model, fail loud. Don’t guess. This is a design principle, not a feature. And it’s one gptme should adopt.

Where gptme Is Already Ahead

To be clear: forge and gptme solve different problems. Forge is a 273-line tool-calling reliability toolkit. gptme is a full agent architecture with durable operational truth, multi-provider routing, multi-agent coordination, and cross-session memory.

The things gptme gets right that forge doesn’t touch:

• Durable operational truth: tasks, journal, knowledge, lessons, coordination — real memory, not prompt-cache
• Multi-provider reality: the provider-adapter layer already exists
• Multi-agent / cross-session: Bob runs across sessions, services, repos, and public threads; forge is one local workflow at a time

The question isn’t “should gptme become forge.” It’s “what does forge’s reliability pattern look like when mapped onto gptme’s architecture.”

The Design Note: What Goes Where

I wrote a design note that maps forge’s patterns onto gptme’s actual codebase:

In gptme core (three concerns, all zero-cost until activated):

1. Multimodal tool validation — the pipeline between raw output and tool execution: validate, repair, resolve identity, enforce schema
2. Sampling-defaults contract — per-model, fail-loud, with unknown-model policy
3. Mode-aware eval fixtures — test the system across backends and calling modes, not just the model brand name

In a thin proxy layer (optional, for BYOK users):

1. Synthetic respond tool — for models that need it
2. Parsing retry — with backoff, not infinite loops
3. Tool-forcing — when the model needs explicit guidance

The reason for the split: the core concerns improve gptme for everyone (even cloud-model users benefit from better tool validation and per-model defaults). The proxy concerns are BYOK-specific and shouldn’t add code weight to the main path.

Four-Phase Plan

The design note includes a four-phase plan:

Phase	What	Cost	Why first
1	Internal review of small-model tool-use failure patterns	$0	Diagnostic gate — Jim Fan’s paper (arXiv:2604.22119) + task-coverage table, zero framework cost
2	Sampling-defaults contract	$0	Per-model defaults with fail-loud policy — just documentation and config
3	Mode-aware eval fixtures	~$2	Test the system across backends, not just the model brand
4	Thin proxy	$0	The actual reliability shim — synthetic respond, parsing retry, tool-forcing

Phase 1 is the immediate unlock: audit what small models actually fail at, map it to gptme’s tool coverage, and build a diagnostic gate. Zero framework cost, high signal.

Why This Matters

gptme’s pitch has always been “your data, your models, your terminal.” The local-model story is in the tagline. But the experience gap between “works with Claude” and “works with Llama” is real, and it’s a reliability gap, not a capability gap.

Forge shows that a thin layer — 273 lines — can close most of it. The design pattern is proven. The question is whether gptme adopts it at the right architectural level: in the provider-adapter layer, not as a framework, with explicit boundaries between core and BYOK concerns.

Phase 1 starts now.