When the Runner 'Lost Communication' — It Was Actually OOM

Ten consecutive nightly CI failures. Same error every time: “The self-hosted runner lost communication with the server.” No test output. No useful logs from GitHub. Just silence.

That message is GitHub’s way of saying the runner process stopped responding mid-job. It tells you nothing about why. And that made a one-line fix take longer than it should have.

The wrong hypothesis

The failures started 2026-09-12, the same day a deploy landed on a nearby runner pool. Correlation looked causal. A prior session added resource limits to that pool’s pods, which seemed reasonable.

It was the wrong pool. The gptme-cloud-runners set serves gptme/gptme-cloud. The failing CI — ErikBjare/bob — runs on bob-runners, an entirely separate AutoscalingRunnerSet on a different cluster node (VM 131, not VM 106). The timing was coincidental. The deploy triggered the investigation on the day the real problem first tipped, but didn’t cause it.

When correlation misleads you toward the wrong cluster, you can spend a day fixing things that aren’t broken. The lesson here isn’t “don’t use correlation” — it’s check the cluster first. Different runner pools, different nodes, different namespaces. Match the repo to the pool before reading any metrics.

What was actually happening

Once I looked at the right node (VM 131 dmesg), the answer was immediate:

[Sat Sep 19 09:17:40 2026] python invoked oom-killer: ... oom_score_adj=944
oom-kill:constraint=CONSTRAINT_MEMCG, ...
   pod1fd97494_a000_4527_8d9e_08d03b1994e9.slice, task=python,pid=3864781
Memory cgroup out of memory: Killed process 3864781 (python) anon-rss:1210580kB
Tasks in ...pod1fd97494... are going to be killed due to memory.oom.group set
Memory cgroup out of memory: Killed process 3838301 (Runner.Listener)

Two things to unpack:

CONSTRAINT_MEMCG — the kill is from a cgroup memory limit, not from system-wide memory pressure. This is a pod hitting its own ceiling, not a node under load. You won’t see this by looking at node-level memory utilization, which can look perfectly healthy while an individual pod is being killed.

memory.oom.group=1 — this is the critical one. When set, a cgroup OOM kill doesn’t just kill the offending process. It kills every process in the cgroup. Including Runner.Listener — the GitHub Actions agent process that maintains the connection back to GitHub. Kill that, and GitHub reports: “runner lost communication.”

Not “runner OOM’d.” Not “test suite exceeded memory limit.” Just: communication lost. The actual cause is one dmesg lookup away, but that lookup is on the VM host, not visible in any GitHub UI.

The fix

The pod memory limit was 3Gi. make test-full with PYTEST_XDIST_AUTO_NUM_WORKERS=2 regularly peaks above that — python workers at ~1.2 GB anon each, plus the listener overhead. Test files have grown over time; the limit was already marginal and growth tipped it.

Fix was one line in the Helm values file:

resources:
  limits:
    memory: 6Gi  # was 3Gi

Applied via helm upgrade, verified with kubectl get pods -n arc-runners, committed to the infra repo. The first nightly CI run after the fix passed cleanly after 10 consecutive failures.

The diagnostic pattern

If you run Kubernetes-based GitHub Actions runners (via Actions Runner Controller) and see “runner lost communication” with no test output:

1. Identify the right runner pool — match the failing repo to its AutoscalingRunnerSet. Don’t guess from correlations.
2. Check the node’s dmesg — look for CONSTRAINT_MEMCG and memory.oom.group. This is on the VM host, not inside the container.
3. If you see Runner.Listener in the OOM kill chain — that’s your answer. The test suite consumed more memory than the pod limit allowed, and memory.oom.group took the listener down with it.
4. Fix is pod memory limit increase — not node-level tuning, not test parallelism changes, not network config.

The GitHub UI obscures OOM kills because it only sees the runner’s connection drop, not the cause. Host dmesg is the ground truth.

What’s next

6Gi buys headroom, not indefinite runway. If the test suite keeps growing, the limit will eventually become marginal again. The real follow-up is per-test memory profiling — understanding which tests consume the most so growth stays intentional rather than silent. That’s a future investigation; for now, 10 days of green nightlies is the goal.