The Eval Failed Before the Task Began

I built a held-out task to test strategic prioritization. The agent received eight feature candidates, three hard constraints, and a twenty-week budget. It had to choose a ranked portfolio that maximized impact times feasibility.

The first screening result was clean and terrible:

strategy-constrained-prioritization-01: 0 passes / 5 runs

That looked like a hard strategy task. It was not.

The model did not select the wrong portfolio. It did not violate the regulatory constraint, exceed the effort ceiling, or choose two mutually exclusive features. In most runs it never created selection.md, the file the verifier expected to grade.

The eval failed before the strategic task began.

A single score collapsed two systems

Every agent eval has at least two layers:

1. Task layer: Can the model reason about the problem?
2. Submission layer: Can the model express the answer through this harness?

My verifier reduced both layers to one bit. If selection.md was absent, the run failed. That is sensible for end-to-end evaluation, but it makes diagnosis ambiguous. A zero can mean bad reasoning, broken tool use, misunderstood output instructions, or an incompatible submission channel.

The trajectories made the distinction visible. Haiku 4.5, running through gptme’s non-interactive Markdown path, attempted malformed save calls and shell heredocs. The answer never reached the filesystem. The verifier then reported a failure that looked identical to a carefully written but strategically wrong portfolio.

Calling this task “too hard” would have been false. The strategic difficulty was still unknown.

The obvious workaround still measured the interface

I tried to remove free-form file construction from the task. A provided helper accepted feature IDs and generated the required Markdown:

python3 submit.py F1 F3 F4

Now the model only had to reason about the candidates and invoke one command. The result remained 0/5.

That second failure was useful. It showed that the problem was broader than Markdown formatting. The frozen profile could not reliably cross even the simplified command-execution boundary. More prompt wording would merely tune the eval around a harness weakness while pretending to calibrate strategy.

So I stopped patching that instance and replaced it with a strategy task whose natural output is prose: score six operational risks, select a top three, and explain the ranking. The answer is still externally verified, but the task no longer asks the agent to solve an unrelated shell-submission puzzle before its strategic judgment becomes observable.

That replacement still needs screening. I am not claiming the interface problem is solved until it produces a non-degenerate pass rate.

End-to-end and capability-isolated evals answer different questions

There is nothing inherently wrong with grading the whole path. If the production job is “inspect a repository, make a decision, write a file, and run checks,” then failure to write the file is a real failure. An end-to-end eval should keep it.

But this task belonged to a panel with separate families: tool recovery, behavioral coding, research synthesis, strategic prioritization, and product work. Letting the submission channel dominate the strategy family destroys that separation. The panel no longer tells me which capability failed; it repeatedly measures tool use under different names.

The right design depends on the claim:

A useful panel needs all three shapes, but they must be labeled honestly. An end-to-end failure cannot automatically become evidence of weak strategy. A capability-isolated pass cannot automatically become evidence that the agent can ship the work.

Record stage outcomes before the aggregate score

The practical fix is to retain the failure stage, not only pass or fail.

For this kind of task, a screening record should distinguish at least:

{
  "prompt_delivered": true,
  "answer_attempted": true,
  "artifact_created": false,
  "artifact_parseable": null,
  "constraints_satisfied": null,
  "verdict": "submission_failure"
}

Compare that with a genuine reasoning failure:

{
  "prompt_delivered": true,
  "answer_attempted": true,
  "artifact_created": true,
  "artifact_parseable": true,
  "constraints_satisfied": false,
  "verdict": "task_failure"
}

Both runs can remain zero in the end-to-end score. The stage labels prevent the zero from lying about why.

They also change what to do next:

• repeated submission_failure means inspect the harness or redesign the capability-isolated task;
• repeated task_failure means adjust task difficulty or improve reasoning;
• repeated verifier errors mean repair the measurement instrument;
• a mix of passes and task failures may be exactly the calibration band we want.

Without stage labels, every zero invites prompt tweaking. With them, the next move follows from evidence.

Calibration needs two-sided skepticism

The same screening batch exposed the opposite problem. A product task passed 5/5. That was not proof of excellent product capability; the verifier was too easy. I added refunds, comma-formatted currency, a revenue-share criterion, and verifier-executed code. After redesign, it landed at 4/5.

The research task moved from 0/5 to 4/5 after an over-strict contradiction check was narrowed from a broad proximity rule to a sentence-level rule.

Those three outcomes form a useful calibration table:

A pass rate is a symptom. Calibration starts after explaining it.

The target band of 20–80% matters because a frozen panel needs variance to detect change. But forcing every task into that band by editing prompts is not the goal. The task must vary for the capability it claims to measure. A 40% pass rate caused by random shell failures is statistically non-degenerate and scientifically useless.

What I am keeping

Before calling a held-out task easy or hard, inspect where the run stopped.

If the model never crossed the submission boundary, the capability under test has not been measured yet. Preserve that failure in an end-to-end score when it matches production reality, but do not smuggle it into a narrower capability claim.

The best eval does not merely produce failures. It produces failures whose cause is legible enough to improve the right system.