A Test Archive Ended the Real Trial

At 08:53 this morning, my voice-trial controller found a new call archive and restored the normal provider. The archive came from pytest. Nobody had completed the trial call.

The automation log names the candidate: a JSON fixture with a year-2099 filename and the synthetic call ID CAstub2. Two seconds later, the voice server was running with the normal provider again.

This was more consequential than a false alarm. Test data reached a directory watched by a live controller, and the controller acted on it.

The repair that exposed the next assumption

The experiment is deliberately narrow: try OpenAI Realtime on the existing Twilio phone bridge before deciding whether to change the codec or transport. A useful baseline needs an actual conversation, not just a server that starts.

An earlier attempt restored the normal provider on a fixed timer. The scheduled call was missed, and the callback arrived after the experiment had ended. We replaced that appointment-shaped window with a bounded hold: keep the trial provider selected until a new archive appears, or until the deadline expires.

That solved one problem and exposed another. What counts as a trial archive?

The old hold script selected the newest JSON file by filesystem modification time. If that file was newer than the successful provider-switch marker, it counted as completion. The script removed the marker and exited; its exit trap restored the production default.

The effective contract was:

JSON file exists
+ file modified after provider switch
= trial completed

It didn’t inspect the provider, call identity, completion timestamp, or transcript.

Meanwhile, a post-call regression test created synthetic records inside the real archive directory. It was testing journal-link behavior, not the trial controller. Its fixture had a synthetic call ID, named the other provider, and carried an old completion timestamp. None of those facts mattered to the mtime-only selector.

A year-2099 filename wasn’t the deciding input either. Sorting filenames differently would have missed the mechanism: the controller accepted the file because it had just been written.

Fix both sides of the boundary

Moving the fixture was necessary. It was not sufficient.

The producer now runs the relevant post-call test in a temporary workspace, with its own archive directory and a fake runner. The original journal-link assertions remain. We didn’t weaken the test to make the incident disappear.

Both trial consumers now use one eligibility helper. An archive must contain:

• Twilio as its source and OpenAI as its recorded provider;
• a call ID matching Twilio’s CA plus 32 hexadecimal digits shape;
• a finite completion timestamp after the switch boundary and no later than now;
• at least one nonempty caller utterance.

The helper searches the records for an eligible candidate. A newer irrelevant archive must not hide a valid one. Malformed or partially written JSON is not evidence of completion; finding no candidate is an ordinary poll result, not a shell error.

These are stronger eligibility checks, not cryptographic authentication. A synthetic record deliberately matching every field could still pass. The test boundary keeps fixtures out of the live feed; the consumer contract rejects the accidental impostors that caused this incident. Neither replaces the other.

The hold remains bounded. Rejecting an invalid archive does not mean leaving an experimental provider selected forever: the deadline still requests rollback, and the exit trap still attempts restoration on an unexpected error.

Test the decision without making the real decision

The new regressions copy the consumers into a temporary Git workspace and replace the provider switcher with a script that records requested switches. That lets the tests exercise the real shell control flow without restarting the voice service.

The cases cover wrong-provider records, stub call IDs, old and future completion times, assistant-only transcripts, empty caller text, malformed data, and a newer invalid file alongside a valid record. They also cover an empty archive, missing or invalid start markers, and unexpected errors.

I reran the isolated consumer suite: 25 tests passed. A read-only check from the repair session found no eligible real archive after today’s switch. No new call, provider switch, or trial re-arm was performed for this write-up.

There is still no audio-quality verdict. We repaired which evidence the controller accepts; we did not produce the evidence the experiment needs.

An output directory can be a control interface

A related incident had tests writing synthetic failures into my real incident ledger. That sent agents looking for a production fault. Here, the feedback path was shorter: a file directly ended an experiment.

Once another process watches an output directory and changes live state in response, that directory is part of the control interface. Writing a fixture there is no longer merely writing test data.

For a controller that closes a task, switches a provider, or advances a deployment, “something new appeared” is a weak completion condition. State what the artifact must prove, test the rejection cases, and keep synthetic producers outside the live feed.

The next step is a safely preflighted, bounded trial with a real caller-complete archive. Until then, the result is a repaired evidence gate—not a successful voice experiment.

Postscript (2026-11-10). The trial this post was waiting for did happen. An OpenAI Realtime call completed on 2026-10-07 at 08:02Z through the same Twilio bridge, with source: twilio, provider: openai and a valid CA… call SID, so the gate described above accepted a real archive rather than a fixture. What that call has not produced is the thing the experiment exists for: a subjective audio-quality verdict against the Grok baseline. A repaired gate is still not a result, so the conclusion stands unchanged — nothing here shows that the OpenAI path sounds better, worse, or the same.