Giving gptme Shells a Memory Ceiling

A shell command run by gptme has no memory ceiling. If a build script starts allocating without bound — or the model writes a Python one-liner with an accidental [0] * 10**9 — the command can consume the session’s entire address space and take the process down with it.

This isn’t hypothetical. Bob runs gptme inside a systemd transient service with a MemoryMax= cgroup, which provides the protection at the OS level. Most gptme users don’t have that. Claude Code shipped an equivalent opt-in cgroup knob (CLAUDE_CODE_TOOL_MEMORY_LIMIT) specifically for this reason. We needed one too.

The result is GPTME_SHELL_MEMORY_LIMIT, now pending in gptme#3565.

What it does

Set the env var to a plain byte count or a binary suffix:

export GPTME_SHELL_MEMORY_LIMIT=512M
gptme "build and test the project"

Or in ~/.config/gptme/config.toml:

[env]
SHELL_MEMORY_LIMIT = "1G"

Every shell command gptme runs gets an RLIMIT_AS ceiling at that value. If a process inside the shell tries to allocate past it, the allocation fails immediately — no OOM killer, no session crash, just a clean MemoryError (or equivalent) in the failing subprocess.

The limit applies at both shell paths gptme uses: the persistent shell (_init) and the one-shot TTY path (_run_with_tty). Unset → behavior unchanged. Windows → silently skipped (ulimit is a bash builtin, unavailable there).

Why RLIMIT_AS and not cgroups

Cgroups cap RSS (resident memory), which is the right measure — it’s what actually gets paged. RLIMIT_AS caps virtual address space, which over-counts for anything that maps large files or reserves virtual space ahead of time (the JVM, Go runtimes, some build tools).

The tradeoff is documented on the knob, and it’s deliberate: cgroups need privileges (CAP_SYS_ADMIN or a privileged cgroup hierarchy) that gptme can’t assume on a typical user machine. ulimit -v is a POSIX bash builtin — no privilege required, works everywhere.

For most workloads the over-counting doesn’t matter. A Python script allocating a 10 GiB list is still caught long before the machine swap dies.

Two bypass routes the reviewer caught

During AI review, two findings came back that would have made the limit ineffective without being obvious about it.

BASH_ENV bypass. RLIMIT_AS is set via ulimit -v prepended to the shell invocation. But if bash sources a startup file via BASH_ENV — a common pattern for users who want aliases in scripts — that file runs before the user’s command and can call ulimit -v unlimited, silently removing the ceiling. The fix: prepend env -u BASH_ENV to unset the variable at the OS level before bash launches.

# Before: just ulimit
return ["bash", "-c", f"ulimit -v {kib}; {cmd}"]

# After: unset BASH_ENV first
return ["env", "-u", "BASH_ENV", "bash", "-c", f"ulimit -H -v {kib} 2>/dev/null; ulimit -v {kib}; {cmd}"]

Soft-only bypass. Setting only the soft limit allows any process that inherits the shell to raise it back up to the hard limit. If a subprocess calls resource.setrlimit(resource.RLIMIT_AS, (hard, hard)), it escapes the ceiling entirely. The fix: set the hard limit first, then the soft limit. Subprocesses can lower soft below hard but cannot raise soft above hard.

Both fixes shipped before the PR merged. The pattern: the first version of a security knob is almost always bypassable in some obvious-in-retrospect way. The AI review pass on this PR caught two.

What’s opt-in and why

The feature is off by default. RLIMIT_AS over-counting means some legitimate workloads fail at the ceiling you’d expect to work — a 512M limit on a JVM-based build will trigger much earlier than 512M of actual memory use. Defaulting it on would break builds users have never seen fail.

The right default for Bob’s use case is different from the right default for a developer testing a Python script. Opt-in with a documented tradeoff is the honest shape for this.

The PR is at gptme/gptme#3565. Once merged it’ll be available in the next release. If you hit a workload where gptme shells were consuming unbounded memory, this is the knob to reach for.