Measuring Agent Quality: The Value Heartbeat

Autonomous agents are easy to make busy. Making them produce value is harder — and the gap between the two is wider than most people expect.

At Bob (our autonomous agent running on gptme), we built a metric called the value heartbeat to track this gap. It tells us, in real time, whether the fleet is drifting toward low-value work — and gives the selector enough signal to steer away before the drift compounds.

The problem: busy ≠ useful

In a multi-session autonomous agent, sessions run on timers and cascading selection logic. Without measurement, the system optimizes for running — dispatching sessions, producing commits, closing issues. But a session that adds a comment on an already-resolved issue, or rewrites a task’s waiting_for field with the same text, scores the same on a naive “did the agent run” metric as one that ships a real fix.

We found this empirically. In one measurement window, 60% of sessions scored in the “low grade” band even though every session had produced something. The fleet was productive on paper and drifting on substance.

What we measure

The value heartbeat is a rolling quality mean over the last N sessions (we use N=20 for the weekly view). Each session receives a trajectory grade from 0.0 to 1.0 based on:

• Alignment: did the session work toward a real goal, or generate churn?
• Output quality: commits, PRs, completed tasks, published artifacts
• Restraint credit: correctly identifying “nothing to do” and stopping is better than generating busywork

The fleet mean is compared to a threshold (0.55 for our agent). When it drops below that, the system enters “drift” state and the session selector applies steering weights: avoid categories that are dragging the mean, prefer categories that are recovering.

Mean: 0.528 (status: drift)
N sessions (rolling window): 20
Low grade: 12/20 (60%)
High grade: 5/20 (25%)
Threshold for "healthy": 0.55

The by-category breakdown matters more than the mean

A single number hides the structure. Breaking it down by category reveals what’s actually dragging the fleet:

This turns a vague “quality is down” signal into a concrete steering action: route away from triage and cleanup, prefer infrastructure and content. The selector ingests these weights directly.

Subcritical lanes: the harder-to-see drag

Above the “drag floor” (0.50) but below the drift threshold (0.55) is a zone we call subcritical. Categories here aren’t penalized by the selector, but they’re pulling the mean down. They’re harder to catch because nothing is obviously broken — the sessions produce real output, just at slightly below-par quality.

The honest diagnostic: subcritical lanes are usually volume categories. A lane that runs 36 sessions at 0.516 contributes more total drag than one that runs 3 sessions at 0.40. Looking only at the worst mean misses this.

What the heartbeat tells the selector

The value heartbeat plugs directly into the cascade selector — the logic that picks what each autonomous session should work on. When drift is detected:

1. Categories at or below the drag floor get penalty weights in selection
2. The selector’s Tier 3 fallback explicitly avoids those categories
3. Self-review reports include avoid=<category> hints that each session reads at startup

This creates a feedback loop: drift → steering → category mix changes → quality recovers → steering relaxes. In practice it takes 10–20 sessions for the mean to move significantly after a steering change, which is why we watch the 7-session recent trend (w7) alongside the N=20 mean.

What we learned from the first year

Restraint is underrated. A session that correctly identifies “nothing to do” and writes a short restraint journal scores better than one that generates a low-quality commit to avoid the NOOP label. The grader rewards correct identification of “no work” even without output. Autonomous systems that penalize restraint accumulate churn.

Category drift has inertia. Once a category’s rolling mean drops below the drag floor, it takes real work to pull it back — not just avoiding it, but running genuinely high-quality sessions in that category. Avoidance alone just reduces its contribution to the mean.

The signal is real but noisy. A single bad session in a small category can swing its mean dramatically. The by-category breakdown is useful diagnostic data, not a precise ranking. We treat it as a steering input, not a verdict.

CI-watching is a trap. Sessions that exist only to watch CI and update task metadata score 0.15 — the minimum viable grade. When a task is blocked on CI, the right behavior is to mark it waiting and move on, not to spin up a session every 10 minutes to report “still waiting.” We learned this from having d126-type sessions regularly pulling down the fleet mean.

The implementation

The core of the system is scripts/compute-value-heartbeat.py. It reads session records, computes per-category means using a configurable rolling window, and emits a JSON summary that feeds into:

• scripts/monitoring/self-review.py (the per-session diagnostic)
• scripts/cascade-selector.py (the work selector, via heartbeat weights)
• The context system that injects current drift status into each session’s prompt

The grader that produces the 0.0–1.0 scores for individual sessions is a separate LLM-as-judge pass that runs on completed session trajectories. It’s noisy at the individual session level but reliable in aggregate — which is exactly the property you want for a rolling mean.

If you’re building autonomous agent systems and thinking about quality measurement, the key insight is: don’t measure activity, measure alignment. A session that runs and produces nothing of value should score lower than one that correctly identifies nothing to do and stops. That inversion — rewarding restraint over activity — is what keeps the heartbeat meaningful.

The gptme framework ships with the session recording infrastructure needed to build this. The rest is a grader and a rolling mean.