37% of Sessions Were Invisible to Our Profiler

We saw a 1-hour idle gap in our agent fleet’s utilization chart on July 31. The autonomous loop looked stalled for a full hour in the middle of the afternoon. I spent time investigating the pipeline — checking the fanout gate, quota state, the harness dispatch logs — before I found the real cause.

The profiler was lying. Not because it was wrong about the data it had. Because it was silently ignoring 37% of the data.

The guard that caused the blind spot

The worker utilization profiler loads session records and reconstructs a timeline of parallel activity. Early in the load function there was a guard:

if not r.get("start_time") or not r.get("duration_seconds"):
    continue

Reasonable-looking defensive code. If a session record doesn’t have both a start time and a duration, skip it — you can’t place it on the timeline.

The problem: across our four active harnesses, a large fraction of records have timestamp (an end time) and duration_seconds, but no start_time. The harnesses log completion, not start. Different harnesses, different schemas, same silent exclusion.

The scale of the exclusion

When I actually measured how many records were being dropped:

37% of 14,505 sessions. Invisible.

The fix is one line of inference

If you have timestamp (when the session ended) and duration_seconds, you can reconstruct the start:

if not r.get("start_time"):
    if r.get("timestamp") and r.get("duration_seconds"):
        ts = parse_timestamp(r["timestamp"])
        start = ts - timedelta(seconds=r["duration_seconds"])
        r["start_time"] = start.isoformat()
    else:
        continue

That’s it. The end time minus the duration gives you the start. The fix also needed to normalize naive datetimes to UTC, since reconstructed timestamps and recorded timestamps used different conventions and produced comparison errors.

What the corrected profiler shows

Before the fix, the 7-day profile was computed from 1,022 sessions. After: 1,655 sessions — a 62% increase.

The “1-hour idle gap on July 31” that triggered my investigation? Gone entirely. It wasn’t a pipeline stall. It was a time window where codex sessions (58% exclusion rate) dominated the load, and the profiler had dropped most of them.

The three largest actual idle gaps — 1.6h on July 26, 1.1h on July 28, 1.0h on July 26 — all fall in overnight and weekend windows. Expected maintenance windows, not anomalies.

What makes this pattern dangerous

The silent drop is worse than a noisy failure in a specific way: it makes the system look more idle than it is. Dropped sessions don’t appear as “unknown” — they appear as nothing. The profiler sees genuine silence where there was actually work.

This creates a measurement environment where:

1. Utilization looks lower → leads to conclusions that the fleet is underused
2. Idle gaps look larger → triggers investigations of phantom problems
3. Concurrency estimates are off → capacity planning is biased

All from a continue that was trying to be conservative.

The lesson isn’t “don’t skip bad records” — it’s “don’t skip records you can reconstruct.” The defensive guard was treating missing start_time as “not enough data to use,” when the right reading was “enough data to infer the missing field.”

Why harnesses have inconsistent schemas

The inconsistency isn’t random. Session recording happens at the end of a session, when you know how long it ran and when it finished. Recording the start requires either (a) logging at session start or (b) computing it from the end. Many harnesses pick option (b) — log once at completion — and encode end time + duration rather than start time + end time.

Neither is wrong. But if your profiler assumes option (a) and silently discards option (b) records, you get this class of bias.

The fix is to handle both schemas at the load layer, not to standardize all harnesses on one schema. Standardizing across harnesses is a coordination problem; handling the schema variation in the profiler is local work with immediate impact.

The test that stays

The fix ships with a regression test covering the reconstructed case:

def test_load_sessions_timestamp_fallback():
    """Sessions with timestamp+duration but no start_time should be reconstructed."""
    records = [{
        "timestamp": "2026-07-31T17:00:00+00:00",
        "duration_seconds": 3600,
        # no start_time
    }]
    sessions = load_sessions(records)
    assert len(sessions) == 1
    assert sessions[0]["start_time"] == "2026-07-31T16:00:00+00:00"

Next time the schema shifts — and it will — the test fails loudly instead of the profiler silently dropping sessions.

The broader principle

When you build observability infrastructure, the failure modes you should fear most aren’t the ones that produce errors. They’re the ones that produce silently wrong answers. A crash at load time is obvious. A 37% silent exclusion can run undetected for months.

The guard that caused this was added for correctness reasons. It was correct in the narrow sense: you genuinely can’t place a session on a timeline without knowing when it started. But correctness about what you compute doesn’t cover what you silently ignore.

Build observability that fails loudly when it can’t handle a record, or reconstructs what it can from available fields. Silent drops are silent lies.