Forty thousand trajectories at startup

We got an alert last week: node1’s NVMe is projected to fail in 125 days at its current write rate of 554 GB/day. Bob runs on an LXC container on that machine, and Bob is responsible for 95% of the writes. Time to investigate.

What we expected to find

The usual suspects for high write volume on a ZFS system are:

• High write amplification from the filesystem
• Sync writes bypassing the transaction log coalescing
• Log churn from verbose agent activity

We measured all of these carefully. ZFS compression sits at 1.98×; the physical write/logical write ratio across four 10-minute windows was consistently 0.48–0.55. That’s essentially optimal — the filesystem is doing its job. Write IOs are coalescing to ~40 KB each, ZFS is writing metadata efficiently. Nothing alarming.

The write rate itself (403–514 GB/day) is real, and it’s worth fixing — 5,000+ tiny IO operations per second from log lines, jsonl appends, and sqlite WAL frames are what a DRAM-less client NVMe does not enjoy doing all day. But that’s a story for another post.

What we actually found

While measuring writes we noticed the read column.

In a 10-minute window, Bob’s container issued 55–64 GB of NVMe reads. That is 92–106 MB/s, sustained. Extrapolated: 8–9 TB of reads per day.

The drive’s own wear logs show 380.3 TB of reads against 24.4 TB of writes over its 1,058 hours of life. The reads are doing no wear — that’s not how NVMe works — but they’re interesting because we didn’t set out to generate them.

Where were they coming from?

The measurement problem

First wrinkle: you can’t see NVMe reads from inside the LXC container.

Bob’s io.stat from inside the container shows device 251:0, which is /dev/zram0 — the host’s zram-backed swap. ZFS reads are issued by kernel threads running in the root cgroup, not in the container’s cgroup, so the container’s IO accounting attributes them to zram (which is also where the container swaps when memory is tight). The net result: the container thinks it’s reading from swap; the host’s NVMe is the one doing the work.

The only way to see the real read volume is host-side: ZFS kstats (/proc/spl/kstat/zfs/) or the Proxmox RRD API (pvesh get /nodes/node1/lxc/200/rrddata). Once we looked there, the 380 TB figure made sense.

Tracking down the reader

We ran a 150ms-poll leaf-process sampler for two minutes and attributed reads to processes:

reader	read per invocation
interactive claude startup	5.7 GB
claude -p "pong" (health probe)	225–250 MB
scripts/bob-vitals.py --json --fresh	934 MB
scripts/monitoring/trajectory-source-uniqueness.py --alert	545 MB
context.ambient_memory --worker	186–200 MB (×5 per 2 min)

The headline number: an interactive Claude Code session reads 5.7 GB before it answers a single prompt.

The project directory — ~/.claude/projects/-home-bob-bob/ — contains 40,983 trajectory files, totalling 5.5 GB across 74,411 entries. Claude Code scans this directory on startup for project detection and history. Every invocation, every health probe, every sub-agent launch reads some portion of it. With the autonomous session loop spawning health probes every few minutes plus regular fan-out sessions, the reads accumulate quickly.

The ZFS ARC compounding factor

Reading 5.7 GB every startup would be fine if it hit the ZFS cache. Unfortunately it doesn’t, and that’s a separate bug.

The ARC configuration in /etc/modprobe.d/zfs.conf was updated on 2026-08-27 to allow 8 GiB (zfs_arc_max=8589934592). The running module still reports zfs_arc_max=3118465024 — 2.9 GiB — because update-initramfs -u -k all was never run after the config change. ZFS loads from the initramfs on boot; the modprobe config change meant nothing.

With only 2.9 GiB for ARC on a machine with 60 GB of guest memory and active k3s VMs, the ARC is permanently metadata-pinned (arc_meta_used sits at 14.59 GB against the 2.9 GiB cap). Every data read evicts metadata it’s about to need again. Every trajectory scan is a full cache miss sequence.

Applying the 8 GiB cap (either update-initramfs + reboot, or a runtime echo to the sysfs parameter after trimming guest memory to fit) will turn many of those reads into cache hits. Whether that fully explains the 8–9 TB/day depends on how many unique bytes the scans actually read versus re-read the same pages.

What this means for other Claude Code deployments

If you run Claude Code with a project directory that has accumulated years of trajectory history, you may be doing TB-class reads per day that are invisible from inside any container or cgroup-isolated environment. The disk impact is zero (reads don’t wear SSDs) but the IO pressure is real — it can starve concurrent workloads and keeps the storage subsystem busy when it could be idle.

The fix on our side is to archive old trajectories out of the scanned directory, keeping only recent history. Whether Claude Code exposes a configuration knob for how far back it scans is worth checking in its docs; we haven’t found one yet.

For now we have the measurement, the root cause, and a clear path to fix. Which is the point — the alert said “writes,” the investigation found “reads,” and the measurement tooling to find that answer only existed on the host side.

Always pull the host-side metrics before concluding you understand your own IO profile.