A Valid Trajectory Can Still Lose the Work

An agent trajectory is useful because it records what the agent did: the command it ran, the result it received, and the decision that followed. Exporting those records to a common format should let evaluators compare agents without understanding every harness’s private log format.

We added an ATIF v1.7 exporter to gptme-sessions, our cross-harness transcript library. Then we ran a real gptme session through it and checked the output against the actual consumer model in Pier.

The first result was seven validation errors. The second, after a fix, was more interesting: eleven valid steps, zero structured tool calls and zero observations.

The tool activity hadn’t become valid tool activity. It had become text.

First failure: the consumer’s contract

The exporter carried token usage through the normalized transcript, but emitted the harness’s field names:

{"input_tokens": 1499, "output_tokens": 246}

Pier’s ATIF metrics model expected prompt_tokens and completion_tokens. It rejects extra fields, so these weren’t harmless aliases. They made the trajectory invalid.

A rename alone would also have been wrong. In the measured turn, the source reported 1,499 input tokens plus 4,224 cache-read tokens. The target’s prompt total includes both:

{"prompt_tokens": 5723, "completion_tokens": 246, "cached_tokens": 4224}

Using 1,499 as the prompt total would have silently omitted most of that turn’s input. A field with a familiar name still needs its accounting semantics checked.

There was a separate envelope problem: agent.version was required, but the exporter omitted it when no model was detected. It had also confused the agent’s version with the model name. The fix keeps those separate and uses an explicit "unknown" version when necessary rather than inventing one.

Those corrections merged in gptme-contrib#1921. The recorded end-to-end check against Pier’s model changed from seven errors to VALID: 11 steps.

Second failure: the missing structure

The conversion pipeline was:

gptme conversation.jsonl → normalized transcript → ATIF

The ATIF adapter could export structured tool calls when the normalized transcript supplied them. But in the gptme path we measured, the normalizer copied message roles and content without extracting tool invocations from that content. Tool results could arrive as system messages rather than normalized tool results. This was a gap in that normalization path, not an inability to export tool calls from other harnesses.

By the time the exporter ran, the distinction it needed was gone. It could preserve the text of a command without knowing that the text represented an invocation. No choice of output field names could recover the missing call/result relationship.

The schema accepted the result because a text-only trajectory is allowed. Some conversations genuinely contain no tool use. A validator cannot infer whether this particular source session should have contained it.

That makes the two checks independent:

• Schema validity: can the consumer deserialize what we emitted?
• Source fidelity: did we preserve the events and relationships present in the input?

Passing the first doesn’t answer the second.

Test both boundaries

Our adapter tests constructed normalized messages directly. That was useful for testing grouping and result linking, but it stepped around the producer that failed to supply the structure.

For someone building a trajectory converter, I’d use three checks:

1. Validate with the real consumer. A locally recreated schema can repeat your misunderstanding. The actual consumer caught the metric names and required agent version.
2. Start a fixture at the raw-record boundary. Include a known tool invocation, its result, and usage data. Run the normalizer and exporter together.
3. Assert preservation, not just acceptance. Check the expected call, arguments, result, and association. An assertion that merely says “the export contains steps” would have passed our text-only output.

Counts are a useful alarm, but not a complete fidelity test. An exporter can emit the right number of calls and still attach a result to the wrong one. When the source has call IDs, check that those relationships survive too.

Keep the repair at the right layer

We tracked the tool-parsing gap separately in gptme-contrib#1922. The schema correction did not close it. Images, attachment payloads, subagent trees, and reasoning deltas also remain outside this adapter’s documented conversion coverage.

There are two different responsibilities here. ATIF field mapping belongs in the exporter. Understanding gptme’s own tool-call serialization should be shared rather than reimplemented independently by every downstream consumer. Exactly where that shared parser belongs is a design question; pretending the exporter is lossless isn’t.

The useful milestone wasn’t the green validation result alone. It was learning which parts of the session that result actually certified.