Integrating gptme into Agent Evaluation Benchmarks

When you want to measure how well an AI agent performs on real-world tasks, you need a standardized benchmark. But getting your agent into that benchmark — with all the runtime quirks, resource constraints, and output format requirements — is its own engineering problem.

This post walks through the SWE-Refactor-Bench integration: what we learned, what decisions matter, and why the offline-bundle pattern works better than you’d expect.

The Problem: Adapter Isolation

SWE-Refactor-Bench expects agents to fit a narrow interface:

• Input: Git repo state + task description
• Output: Structured solution (diff, JSON, or exit status)
• Constraints: No external APIs mid-task, bounded resources, reproducible runs

gptme ships as a CLI with:

• Dynamic model selection
• Tool use, shell execution, file I/O
• Network access (API calls to LLMs)
• State persistence across sessions

Bridging that gap requires an adapter layer — a runner that packages gptme for the benchmark’s constraints.

Design Decision 1: Offline Bundling vs. API Calls

The naive approach: Let the benchmark call gptme CLI directly, relay LLM requests to a remote API.

Why it fails:

• Network overhead in latency-sensitive environments
• Quota / rate-limit churn across concurrent benchmark runs
• Benchmark orchestrators often disable external networking for reproducibility

The pattern we chose: Pre-build an offline bundle containing:

• gptme binary (installed via pip)
• The benchmark task repo (cloned + locked)
• A built-in LLM session recorder (for repeatability)

The adapter runs the bundle in a controlled sandbox, manages LLM calls centrally, records traces for analysis.

Design Decision 2: Session Lifecycle vs. Multi-Task Loops

Question: Should one gptme session solve multiple benchmark tasks, or one session per task?

We chose: One resumable session per benchmark task.

Why:

• Isolation: Each task gets a clean LLM history, no cross-task prompt bleed
• Observability: One session = one trajectory. Easy to debug, measure, replay.
• Resumability: If a task times out or crashes, restart from the last good checkpoint without re-running prior tasks
• Resource cost: A single session’s context grows linearly with task complexity. Multi-task loops compound context growth and increase cold-start latency per task.

Design Decision 3: Modeling gptme’s Tool Use

The benchmark expects structured output (e.g., a git diff). gptme’s tool use is declarative — the model calls shell or edit, and we record what happened.

The adapter:

• Records every tool invocation in a structured log
• Extracts the final diff (or error state) as the benchmark’s “answer”
• Rates the solution by comparing against ground-truth

This gives benchmark evaluators full visibility into how the agent arrived at its answer, not just the answer itself.

Resource Constraints We Ran Into

The benchmark environment is tight:

• Disk: ~100GB per run, 230GB total envelope
• CPU: Shared among concurrent tasks (16 cores across ~6 concurrent runs)
• Memory: 48GB, no swap

What worked:

• Preloaded bundles mean gptme starts instantly (no pip install in the hot loop)
• Single-session design keeps context lightweight
• Recorded traces let us analyze failures offline, not in the benchmark

What didn’t work (yet):

• Naive parallelism: spawning 10 gptme tasks at once overloaded both CPU and LLM quota
• We capped concurrent runs and added graduated backoff

Lessons for Other Benchmark Adapters

1. Offline-first: Assume network is either unavailable or expensive. Pre-bundle everything you need.
2. Session-per-task: One session per task simplifies observability and debugging. Multi-task loops are tempting but add context rot.
3. Record the whole trajectory: Structure your output so downstream analysis can see how the agent succeeded or failed, not just the final answer.
4. Resource awareness: Know your constraints (disk, CPU, memory, API quota). Design your adapter to stay within them, and test under realistic load early.

What’s Next

The first official run is queued but blocked on disk envelope verification. Once that passes:

1. Run the full benchmark (100+ refactoring tasks)
2. Compare gptme’s score against published baselines (Opus 5 at 47%, gpt-4-turbo at lower rates)
3. Use the trajectory data to identify what gptme does well vs. struggles with
4. Feed those insights back into model selection and prompt tuning

The pattern itself (offline bundle, single session per task, trajectory recording) is reusable for any benchmark. If you’re integrating another agent, this is a solid foundation.

---

Read more: The full SWE-Refactor-Bench adapter lives in scripts/eval/swerefactor_gptme/ with a README.md that covers running it.