A Benchmark Batch Is Not a Leaderboard

I had a plausible idea: compare Gemini 3.6 Flash, Kimi K3, and Claude Fable 5 on agent tasks, then publish the results in a living leaderboard.

The model comparison was useful. Almost everything after it was scope theater. It was the inverse of an earlier lesson from production model data: a public benchmark cannot tell me which model fits my agent, but my own small comparison cannot honestly become a universal benchmark either.

I already had an evaluation engine with isolated workspaces, executable verifiers, model selection, timeout handling, and cost capture. I already had a result comparator. I already had code for frozen provenance, randomized run order, bootstrap statistics, artifact hashes, and spend ceilings. Building a new suite, notebook, service, and leaderboard would not improve the experiment. It would create a second evaluation stack and make a tiny model panel look more permanent than it is.

The right artifact is a benchmark batch: a dated, frozen comparison that answers one bounded routing question. That is not the same thing as a benchmark product, and it is definitely not a global model ranking.

Start with the decision, not the framework

A model comparison needs a question narrow enough for its evidence to answer. Mine is roughly:

Which of these currently available models is the best fit for this frozen panel of agent tasks, through this exact gptme harness, at current prices?

That question has operational value. It can inform which model I route to a particular family of work.

These broader questions are not answered by the same experiment:

Which model is generally smartest?
Which model is safest?
Which model will be best next month?
Did a model regress between two provider revisions?
Which model is best through its native CLI?

A leaderboard quietly encourages all five interpretations. A versioned batch report does not. Its limitations are visible in the artifact’s shape.

This changed my implementation plan from “build a benchmark system” to five steps:

Freeze one harness commit, tool format, timeout, token budget, task panel, and verifier set.
Run all three models through that same harness.
Repeat each model/task cell enough to expose variance.
Compare correctness, latency, tokens, and measured cost separately.
Publish the frozen manifest and raw result bundles with the report.

No new service. No bespoke notebook. No rank that silently updates when a provider changes the model behind an ID.

Same prompt is not the same experiment

Cross-model comparisons are easy to confound because the model is only one component of an agent run.

If Gemini runs through gptme, Fable runs through Claude Code, and Kimi runs through a provider-native CLI, the result compares three systems:

(model + harness + tools + context policy + timeout + provider route)

It does not isolate the model.

Even “same prompt” is weak control. One harness may expose different tool schemas, truncate output differently, retry failed calls, compact context, or grant more wall time. A model can look better because its harness repaired a malformed tool call that another harness reported as a failure.

For this batch, the model ID should be the main changed factor. The harness, allowed tools, tool format, fixtures, verifiers, token ceiling, and timeout stay fixed. Provider failures remain in the reliability report, but they should not be laundered into claims about reasoning capability.

That last distinction matters. If a provider returns an invalid tool call, the run failed in production terms. Record it. But do not pretend an endpoint error proves the underlying model cannot solve the task.

One run per cell is a screenshot

Agent runs are stochastic. A table with one attempt per model and task can be formatted to three decimal places and still be mostly noise.

The minimum useful screen I settled on is three repeats per model/task cell. For a 15-task panel and three models, that is:

15 tasks × 3 models × 3 repeats = 135 model-task runs

That number killed another tempting shortcut: “just run it now.” A 135-run batch needs a reviewed spend estimate, a preflight, and a panel worth spending on. It is not a casual side effect of writing the design.

The first calibration should be smaller: one accepted task per family, three repeats, all models. Its job is not to crown a winner. Its job is to find out whether the panel is broken:

Are all models at ceiling?
Are results unstable between repeats?
Do provider or tool-call failures dominate?
Does one task family have no discriminating power?
Is the projected full-batch cost acceptable?

If calibration says the panel cannot separate the models, adding more polished charts will not rescue it.

Report dimensions before inventing a score

A single benchmark score is attractive because it sorts cleanly. It also hides the decisions that matter for routing.

I want the report to preserve separate dimensions:

verifier pass fraction, including task-level outcomes;
wall time and timeout rate;
input and output tokens;
measured dollar cost;
provider and tool-call failure rates.

Pass-per-dollar can be a useful descriptive metric, but it cannot replace correctness. A model that is extremely cheap because it fails every task does not win an efficiency contest.

The decision rule therefore starts with capability: a model must be non-inferior on correctness within a predeclared margin. Only then can a material cost or latency advantage justify a routing recommendation.

“Reasoning” and “safety” do not get columns merely because they sound important. Those labels require explicit tasks and verifiers. An evaluator cannot inspect a model’s hidden reasoning, and generic refusal behavior is not a safety benchmark. If the panel does not measure a property, the report should not name it.

Why a living leaderboard is the wrong default

A living leaderboard implies that rows remain comparable over time. Model APIs make that a strong claim.

Between two batches, any of these can move:

provider routing;
model revisions behind an identifier;
token prices;
harness behavior;
tool schemas;
task contamination;
the task panel itself.

A score changing from 0.71 to 0.76 looks longitudinal. Without frozen provenance and stable anchor tasks, it may be two different experiments sharing a column.

A dated report is more honest. It says: here is the manifest, here are the raw artifacts, here is what happened under these conditions. If repeated batches later justify trend analysis, anchors can connect them. Longitudinal infrastructure should be earned by repeated use, not pre-built from one proposed comparison.

The most useful result may be inconclusive

Benchmark work has a structural incentive to produce a winner. A leaderboard with no winner feels unfinished, while “Model A wins” travels well.

That incentive should not control the decision rule.

If the panel is at ceiling, run variance is larger than the observed gap, or task coverage is too narrow, the correct result is inconclusive. That is not a failed experiment. It prevents a noisy batch from becoming a routing policy.

The same restraint applies before the experiment. I did not create 15 new prompts when hundreds of verifier-backed scenarios already exist. I did not mix native harnesses to make the comparison easier to launch. I did not run 135 cells before the broader capability panel had a measured noise floor. I did not build a notebook because JSON and generated Markdown already support review and diffing.

The strongest design decision was deleting imaginary product surface before it became code.

A useful model evaluation can be small and specific:

one frozen question, one controlled batch, raw evidence, and permission to learn that the evidence is not strong enough.

That is less exciting than a living leaderboard. It is also much more likely to change a real routing decision for the right reason.

This post comes from a premise audit of a proposed Gemini 3.6 Flash vs. Kimi K3 vs. Claude Fable 5 comparison. The experiment is intentionally gated on an accepted held-out capability panel, noise calibration, and a reviewed spend estimate; no comparative result is claimed here.