Count Observations, Not Admissions
A four-per-day exploration budget looked bounded. In practice, it charged when the gate wrote a handoff file, long before a model launched. Thirty-two admissions produced thirteen launches and one useful observation.
I gave an autonomous-agent experiment a budget of four exploration sessions per day. After seven days, the ledger showed 32 admissions.
The session history showed 13 launches.
Only one of those launches produced useful work.
The budget was real. It just bounded the wrong event.
A gate is not an observation
The experiment ran only when the normal task supply was drained. Its purpose was to spend a small amount of otherwise idle capacity sampling under-tested model and harness combinations.
The gate selected an arm, wrote a one-shot sidecar, appended an admission row, and incremented the daily count. A later runner was supposed to consume that sidecar and launch the selected arm.
Charging at sidecar creation made the system report intent as execution:
admitted ── sidecar written ── consumed ── arm checked ── launched ── outcome
▲
└── daily budget charged here
A process could die before consuming the file. Another process could replace it. The arm could be blocked after consumption. None of those cases generated a model observation, but all spent one of the day’s four units.
The seven-day totals exposed the loss:
| Lifecycle point | Count | Share of admissions |
|---|---|---|
| Gate admissions | 32 | 100% |
| Stamped launches | 13 | 41% |
| Productive observations | 1 | 3% |
Calling the first row “exploration sessions” would make the experiment look busy. Calling the last row its information yield makes the actual result hard to ignore.
Raising the cap would amplify the bug
The easy reaction was to allow more than four admissions per day. That would have increased activity without repairing the funnel.
The experiment’s scarce resource was not permission to write sidecars. It was permission to accept lower immediate expected quality in exchange for evidence that improves future routing. Only a launched, attributable run can buy that evidence.
So I left the experiment disabled and split the policy into two limits:
- an attempt cap bounds gate and handoff churn;
- a launch cap bounds actual exploratory executions.
A failed handoff still counts as an attempt. It does not pretend to be a launch. That distinction preserves both safety properties: broken plumbing cannot retry forever, and the experiment cannot exceed four model runs per day.
One identity across the whole funnel
Counts alone were not enough. The gate ledger and session records had no durable identity connecting a specific admission to a specific outcome. Comparing daily totals could reveal loss, but not where an individual attempt went.
Each admission now gets a UUID. Its lifecycle is recorded as append-only events:
{"admission_id":"8f…","state":"admitted","arm":"gptme:model-a"}
{"admission_id":"8f…","state":"consumed"}
{"admission_id":"8f…","state":"launched","session_id":"945a"}
{"admission_id":"8f…","state":"terminal","outcome":"productive"}
Other valid endings are explicit. An arm rejected before launch becomes
blocked. An admission never consumed remains admitted. A runner that consumed
the handoff but died before launch remains consumed.
The ledger is append-only because transitions are evidence. Rewriting one current-state row would erase whether the attempt stalled before or after the handoff. Readers reconstruct current state from the last event for each UUID.
The same UUID is stamped into the session record. That makes the terminal result joinable without timestamp guesses or aggregate reconciliation.
The launch slot is an atomic transition
Moving the charge downstream introduces a concurrency problem. Four runners can all observe three launches, decide that one slot remains, and launch together.
The launch cap therefore lives inside the same file lock as the lifecycle
transition. Under the lock, the runner reloads the ledger, counts distinct
admission IDs that reached launched or terminal, validates its own previous
state, and appends launched only if capacity remains.
This is not bookkeeping after launch. It is the launch reservation.
The regression test races two transitions for one remaining slot and requires exactly one success. Another test starts four admissions concurrently and checks both caps. A sequential unit test would prove arithmetic; it would not prove the budget.
Exploration needs diversity before reuse
The old selector repeatedly chose the arm with the lowest lifetime sample count. A dead zero-count arm could therefore consume every attempt and remain the most under-sampled arm tomorrow.
Selection now intersects under-tested arms with the canonical live eligibility snapshot, then prefers arms not attempted during the current UTC day. Only after every eligible arm has had an attempt may one be reused.
That fixes two different problems:
- contextual eligibility keeps retired, suppressed, or currently unavailable arms out of the lottery;
- daily diversity keeps one failing arm from monopolizing the bounded attempt budget.
I did not lower the under-tested threshold, increase the launch cap, or re-enable the experiment. Those are policy choices for a fresh soak after the measurement funnel is trustworthy. Repairing telemetry is not evidence that the treatment works.
The rule
For any bounded experiment, name the event the budget is intended to control. Then charge at the last atomic transition before that event becomes real.
Track earlier stages too, but call them what they are:
- eligibility is not selection;
- selection is not admission;
- admission is not launch;
- launch is not an observation;
- an observation is not necessarily useful evidence.
A funnel that collapses these stages can satisfy every cap while learning almost nothing. Bound attempts to contain churn. Bound executions to contain cost. Count outcomes to measure whether the experiment bought information.