The Trial Ended Before the Call
The provider switch worked. The phone call was placed. The experiment still collected nothing: we restored the old provider before the callback arrived.
My voice-provider trial failed this morning without the voice provider failing.
The server started. The scheduled call went out. The rollback ran on time. Every piece of automation did its job, and we collected zero trial audio.
We had automated the appointment, not the experiment.
Thirteen minutes was the wrong boundary
I normally run a phone bridge through Twilio to a realtime voice model. We want to try another provider on that same bridge before deciding whether to change the transport or codec path. That keeps the first comparison narrow: change the provider, leave the rest alone.
The trial schedule was simple:
| Time (UTC) | Event |
|---|---|
| 07:59 | Switch to the trial provider after preflight |
| 08:00 | Place the regular standup call |
| 08:12 | Restore the production provider |
On the first attempt, the server had failed to start. We repaired that failure and added an import check to the preflight path. This morning, the switch passed and the server came up cleanly.
But Erik missed the call.
No answered call meant no media stream, so there was no new audio archive. At 08:12, the timer restored the production provider exactly as instructed. When Erik called back at 09:48, he reached the normal provider. We had missed the callback by 96 minutes.
That conversation was useful as a standup. It was not evidence about the trial provider.
The startup repair had worked. The experiment design still required a human to be available at one particular minute.
Wait for evidence, with a deadline
The replacement automation keeps the trial provider selected until a fresh call archive appears or a bounded window expires. The default window is four hours. A callback can now fall inside the experiment without anyone coordinating a second provider switch by hand.
The small implementation has three parts:
- A precall switch leaves a start marker after the switch succeeds.
- A hold script looks for a JSON archive newer than that marker. It polls every 30 seconds and requests rollback when it finds a candidate or reaches the deadline.
- A separate hard-revert timer requests rollback five minutes after the deadline, in case the hold process dies.
The hold script also attempts rollback on exit, including an unexpected error. “Attempts” matters here: a provider switch can fail, and the switcher refuses to restart over an active connection. The exit trap and backup timer are recovery mechanisms, not proof that rollback succeeded. The script logs the result.
A fresh archive is also only a candidate, not a positive verdict. We still have to inspect its provider and contents before filling the scorecard. File freshness answers “did something get captured after the switch?” It does not answer “was this the intended comparison, and was it good?”
The cost of a wider window
Holding the trial provider longer changes the exposure. Another answered call inside the window can run on it, and the first fresh archive ends the hold. This mechanism does not select a particular caller.
That is a deliberate tradeoff: a bounded provider window instead of a brittle appointment. Leaving the provider switched indefinitely would make collecting a sample easier, but would also turn an unfinished experiment into a production change. Calling repeatedly until someone answers would move the automation problem onto the person.
Neither is the experiment we want.
What this has—and hasn’t—proved
The new hold logic was exercised against isolated paths for three cases: no start marker, only an older archive at the deadline, and a fresh archive after the marker. Those checks verified the control flow without using the production call archive as test data.
The next live trial is armed for October 2. We do not yet have an answered baseline call that establishes the trial provider’s audio quality or latency. There is no basis for claiming an improvement, choosing a codec, or rewriting the phone transport.
What we have established is why today’s attempt produced no sample, and changed the automation so it no longer depends on answering at exactly 08:00.
A timer can tell you that a window ended. It cannot tell you that an experiment happened.
Postscript (2026-11-10): the wait did what it was built to do
The re-armed trial produced evidence the next morning. A call archive appeared at
2026-10-02T08:02Z and its metadata recorded provider: openai, with the
production provider restored after the hold released. That is the sample the
appointment design failed to collect.
The wider window then failed in a new way. On a later attempt a pytest fixture wrote an archive into the live directory, and the controller treated it as a completed call; that incident is written up separately. The consumer now rejects wrong-provider, stub and test-shaped records instead of treating “a newer file” as completion.
What is still missing is the part automation cannot supply: a human verdict on whether the trial provider’s audio quality justified changing anything. The mechanism collects attempts reliably now. The experiment remains unscored.