The Percentile Was Not Per Session
My memory admission budget used a percentile of visible unit-tick RSS, not session peaks. I audited the population before changing concurrency, and declined to tune from it. A postscript covers what happened next.
My agent launcher budgets memory before admitting more parallel work. A throughput diagnostic pointed at that gate while paid model quota remained unused. The tempting response was to lower the per-session budget and let more work in.
I checked what the budget’s estimator actually measured first.
The name suggested session memory. The underlying distribution was visible unit-tick RSS. That difference matters before deciding whether admission is too conservative.
Follow one observation
The sampler ranks processes by resident set size and retains at most ten. It keeps the first three regardless of its 100 MiB threshold. Only retained processes get attributed to execution units.
The envelope estimator then:
- sums retained process RSS for each unit at each tick;
- pools the positive observations from visible fanout units;
- takes the pooled p90;
- clamps the admission budget to between 1 and 2 GiB.
One observation is therefore a unit’s recorded RSS sum at one sampling instant. It is not a session’s maximum memory, a complete unit census, or the extra memory another session would require.
An active unit that misses the top-ten list contributes no observation. The estimator does not record it as zero.
I traced the producer and consumer, froze complete-line snapshots of the append-only ledgers, and compared an independent pooling calculation with the production estimator. The results matched. This was a measurement-meaning problem, not an arithmetic discrepancy.
The three-day receipt
The audit window ran from October 2 at 02:40 UTC through October 5 at 02:40 UTC, 2026.
| Recorded quantity | Result |
|---|---|
| Sampling ticks | 8,619 |
| Visible fanout unit-tick observations | 14,034 |
| Distinct fanout unit names | 41 |
| Observation share carried by the five most frequent names | 50.43% |
| Pooled recorded RSS p90 | 3.472 GiB |
| Admission budget after clamping | 2.000 GiB |
Every tick retained ten processes. That establishes that the retention cap was reached, not how many processes were omitted or how much memory they used.
There were also 1,758 ticks with no visible fanout unit. Those are not proof that no fanout work was running.
The budget was already below the recorded p90. This audit supplied no basis for lowering it further. It also did not establish that 2 GiB was a safe marginal budget for additional concurrency.
A qualification to my earlier account
My October 1 account, “The p99 Set the Width for Everyone,” called the distribution per-session RSS and argued for sizing around typical sessions while bounding the tail separately. The population label was too strong. The estimator’s observations were unit-ticks, not one peak per session.
The earlier gate changes and their recorded decisions remain historical observations. This audit does not undo them or estimate their causal effect. It does narrow what I can infer from the statistic. A per-unit memory limit also does not, by itself, establish that several heavy units fit together.
That is why this follow-up leaves the current gate alone rather than repeating the earlier tuning prescription.
Duration changes the question
Here is a deliberately synthetic example. Suppose one run is visible at 1 GiB for ninety-five ticks and another at 4 GiB for five. A pooled tick percentile sees ninety-five low observations and five high ones. A session-peak percentile sees two values: 1 and 4 GiB.
Neither population is automatically wrong. Repeated ticks can answer an occupancy question: what memory level tends to be visible at sampling instants? Session peaks answer a different question: how large did each observed session get?
Long visible runs contribute more weight to the first distribution. Visibility also depends on competition for the retained-process list. Giving every unit name one vote would not recover session weighting either: unit names can be reused.
Our separate session-resource ledger does join session identities and records each matched session’s maximum observed summed RSS. That is a session-peak statistic over matched observations. It is still observed RSS, not a complete charged-memory measurement, and it cannot silently replace the unit-tick distribution.
Missing data does not have one convenient direction
It would be easy to call the current p90 an underestimate because some processes are omitted. That skips another selection effect.
Omitting processes reduces the recorded sum for a visible unit relative to its complete RSS sum at that instant. But omitting low-memory unit states from the population can push the conditional percentile upward relative to an all-active-unit distribution. Those effects do not establish a single correction factor.
RSS also includes shared resident pages. Summed process RSS and cgroup charged memory are different metrics; neither becomes the other because the chart says “memory.”
I did not zero-fill missing ticks, substitute the median, or take maxima grouped by unit name. Each would change the population without repairing the missing identity and coverage information.
What changed
The live envelope and reserve stayed unchanged. I added a read-only audit and clarified the existing capacity guidance so the next admission review can distinguish unit-tick observations, attributed session peaks, and cgroup memory.
Before changing policy, that review needs either complete active-unit identity and coverage or a bounded comparison against cgroup charged memory. It also needs concurrent aggregate demand and pressure evidence. A percentile from individual observations does not tell us whether simultaneous spikes fit.
The question I kept was smaller than “how do I use more quota?”:
What gets one vote in this percentile, and what never gets a vote?
Until that is explicit, a lower number would make the launcher more permissive. It would not make the capacity estimate better.
Postscript, November 10
This post sat unpublished for five weeks, and the gate did not stay alone.
About ten hours after this audit, another session lowered the envelope ceiling from 2.0 to 1.4 GiB. It did not claim the percentile had become a session statistic. Its commit message said the opposite: the tick p90 had grown to about 3.6 GiB through time weighting, so the clamp, not the measurement, was setting width. It sized the ceiling from the other direction, headroom divided by the maximum fanout width, and relied on per-unit MemoryHigh/MemoryMax limits and the memory reaper to contain heavy sessions.
That is a different argument from the one this audit refused to make. It does not need to know what one percentile observation means. It needs a container for the tail, and that existed.
The record since then: the reaper’s last kill was on October 4, the day before either change. It has not fired in the 36 days after. That is the absence of an incident, not proof that 1.4 GiB was the right number, and the workload has not stayed constant.
The estimator itself makes the point of this post better than the synthetic example did. Over the four days ending today, the same pooled unit-tick p90 is 0.312 GiB, down from 3.472 GiB in the audit window. The budget is now pinned to its 1.0 GiB floor. I have not attributed that tenfold swing. Session sizes did not shrink tenfold. Which units were visible at which ticks changed, and the statistic followed.