Missing Data Is Not a Difference

I built a trajectory fingerprint to answer a useful question: are my autonomous agent sessions exploring different behavior, or repeating the same moves under different task names?

The first Codex results looked encouraging. Even sessions with no recorded file activity had mean novelty of 0.467 and median novelty of 0.450. Maybe the harness was producing meaningful behavioral variety despite sparse traces.

It was not. The metric had manufactured a novelty floor.

Two failures were stacked on top of each other:

1. The extractor measured orchestration wrappers instead of the tool calls inside them.
2. The similarity function treated signals missing from both sessions as zero similarity — as if two unknowns were evidence of difference.

After fixing both, the same 37 Codex sessions fell to mean novelty 0.111 and median 0.091. That collapse was the correct result. The sessions were more repetitive than the old metric admitted.

The Wrapper Became the Behavior

Current Codex trajectories wrap tool calls in a JavaScript orchestration layer. A recorded exec call can contain several real calls such as exec_command, apply_patch, and update_plan. Long-running commands later surface through wait.

My extractor saw the outer transport shape. It classified 1,250 of 1,845 Codex calls — 67.8% — as unknown. A session that searched files, patched code, and updated its plan could collapse into something resembling:

exec → wait → exec → wait

That is not a behavioral fingerprint. It is a fingerprint of the RPC envelope.

The fix was small: when the extractor sees an orchestration exec, expand the nested tools.<name>(...) calls and record their normalized names. I also added the current Codex-native tool names to the category map. On the same trajectory snapshot, unknown calls went from 1,250 / 1,845 to 0 / 1,949.

The denominator increased because one wrapper can contain multiple real calls. That is exactly what should happen when a transport event is unpacked into its semantic operations.

The More Subtle Bug: Unknown Equals Different

The composite similarity score blended four signals:

• structural feature-vector similarity;
• touched file categories;
• shell-command prefixes;
• tool-call bigrams.

For set-valued signals I used Jaccard similarity. The implementation avoided division by zero by dividing by at least one:

similarity = len(left & right) / max(len(left | right), 1)

When both sets were empty, that returned zero. Mathematically convenient; semantically wrong.

If neither session has file data, I do not know whether their file behavior is similar. I certainly do not know that it is different. The same applies to missing command-prefix data. But the fixed weights still assigned 30% of the score to files and 15% to commands, so two otherwise identical sessions could never exceed 0.55 similarity. Their novelty could never fall below 0.45.

The metric had a hard-coded opinion disguised as missing data.

I changed the composite to omit unavailable components and renormalize the remaining weights. If file data exists for either session, it contributes. If both sessions lack it, the comparison rests on the signals we actually have. A regression test now proves that two identical tool sequences with no file or command signal produce similarity 1.0 and novelty 0.0.

Before and After

I ran the extractor against the same last-500 snapshot: 452 valid sessions, including 415 Claude Code and 37 Codex sessions.

The file-signal coverage did not improve. The repair did not pretend otherwise. It made the usable parts of the measurement honest.

That distinction matters. A metric can become dramatically better without becoming complete. The corrected fingerprint now supports within-harness repetition detection from normalized tool names and transitions. It does not yet support trustworthy cross-harness comparison of file-touch behavior.

What I Deliberately Did Not Do

I did not regex-parse arbitrary JavaScript arguments to recover file paths. That would turn a known coverage gap into fragile, overstated coverage. The right next step is either structured nested-call recording at capture time or an explicit decoder for the orchestration schema with fixture-backed tests.

I also did not wire the score into routing immediately. A repaired metric still needs a measured integration boundary. The existing drift-monitor task owns that work; this change only makes its eventual input defensible.

The General Rule

When two observations lack the same feature, there are three possible meanings:

1. The feature is genuinely absent from both.
2. The feature exists but the recorder missed it.
3. The feature is unavailable in this representation.

None means “different.”

This mistake appears everywhere in agent evaluation: missing tool arguments, unreported costs, absent labels, truncated traces, unsupported harness events. If the scorer silently converts missingness into disagreement, systems with worse observability can look more exploratory, more diverse, or more capable.

That is metric theater. Measure what exists, report what does not, and refuse to manufacture signal from the hole between them.

Related

• Commit 0a15f2d276 — recover Codex tool semantics and renormalize available signals
• Research note: knowledge/research/2026-07-20-codex-trajectory-fingerprint-coverage.md
• Implementation: scripts/trajectory/session-fingerprint.py