The Filename That Crashed the Checkpoint

A late bug in gptme’s conversation-checkpoint prototype was hiding in a filename that Python could represent but UTF-8 could not encode.

On POSIX systems, filenames are bytes. Most of those bytes are UTF-8 in practice, but the filesystem does not require that. A repository can contain a perfectly valid path such as this:

raw_path = b"invalid-\xff.txt"

Python still has to expose that path through its string APIs. It does so with the surrogateescape error handler: the undecodable byte becomes a lone surrogate code point. That preserves enough information to recover the original bytes later.

It also creates a value that looks like a normal str until it reaches a strict Unicode boundary.

Our new checkpoint code reached two of them.

The first failure: storage

A conversation checkpoint records the working-tree changes needed for another session to resume the work. Git’s machine-readable status output gives us raw path bytes, which the checkpoint code decodes like this:

def _decode_git_path(value: bytes) -> str:
    return value.decode("utf-8", errors="surrogateescape")

That is the right decoding policy for a lossless internal representation. The problem appeared later, when the checkpoint was written as JSON:

json.dump(checkpoint.to_dict(), output, indent=2, ensure_ascii=False)

ensure_ascii=False normally makes JSON easier for humans to read. Here it asked the strict UTF-8 text stream to encode the lone surrogate. UTF-8 has no valid encoding for it, so saving the checkpoint raised UnicodeEncodeError.

The first fix was small: let JSON escape every non-ASCII code point.

json.dump(checkpoint.to_dict(), output, indent=2, ensure_ascii=True)

The file now contains an ASCII sequence such as invalid-\udcff.txt. The file is valid UTF-8 and Python’s JSON decoder restores the same Python string. The unpaired surrogate is not interoperable JSON, though: other decoders may reject or replace it. A portable cross-language format would encode the original path bytes explicitly, for example as base64. A regression test creates the filename from raw bytes, saves the checkpoint, reloads it, and checks both the path value and the escaped bytes on disk.

That test passed. The checkpoint was durable on disk.

That edge case still was not usable.

The second failure: presentation

The next review pass followed the data one step further. Loading JSON turns the escape back into the original lone surrogate. The resume command then renders the checkpoint as a handoff prompt and sends it to click.echo.

POSIX sys.stdout commonly uses surrogateescape, which can write that value back as its original byte. A CLI cannot assume every output stream has that policy, though. A UTF-8 stream with errors="replace" accepts the restored surrogate and substitutes a replacement such as ?. Only a strict stream raises UnicodeEncodeError. The resume command could still crash at that stricter output boundary:

Git bytes
  -> surrogateescape string
  -> ASCII-safe JSON
  -> surrogateescape string
  -> strict UTF-8 output stream

We had made persistence lossless without making human output safe.

This is the kind of bug that slips through if “round trip” means only save() followed by load(). The artifact survived storage, but the product operation was save() followed later by resume. The second path had its own encoding contract.

Lossless storage and safe display are different contracts

The checkpoint file and the terminal should not apply the same policy.

The stored artifact needs fidelity. If a path contains arbitrary POSIX bytes, the checkpoint should preserve them rather than silently rename or drop the file. surrogateescape plus ASCII JSON escaping gives us that property.

Human-facing output has a different job. A terminal cannot display an invalid UTF-8 byte as the original character because there was no original character. It needs a safe, explicit representation of the path — for example an escaped form — while keeping the stored checkpoint untouched.

That separation already existed for another reason. Resume output strips terminal control sequences before printing so persisted conversation text cannot manipulate the operator’s terminal. Non-UTF-8 path bytes belong to the same architectural boundary: preserve raw meaning in structured state, but make the rendered view safe for the output channel.

Trying to “fix Unicode” globally would be the wrong move. Replacing undecodable bytes during Git parsing would lose path identity. Opening the JSON file with a permissive encoder would produce data other tools may reject. Reconfiguring the process-wide terminal stream would spread a checkpoint-specific edge case into every CLI command.

The narrow design for this Python-owned checkpoint format is stronger:

• parse Git’s byte-oriented protocol without ambiguity;
• retain filesystem identity internally;
• serialize it in ASCII-safe JSON that Python can round-trip;
• convert it to a display-safe form only at the human-output boundary.

Each layer owns one contract.

Test the operation, not the helper pair

The original regression was valuable: before the fix, it proved that one filename could prevent a checkpoint from being created at all. But its final assertion stopped at load_conversation_checkpoint().

The review finding exposed the missing acceptance test. For a resumable checkpoint, the end-to-end assertion is not merely that the Python object can be reconstructed. It is that the supported resume command can render the handoff through the output policies it claims to support, including a strict UTF-8 stream.

That is a broader durability rule:

Stored is an intermediate state. Durable means the next consumer can use it.

A database row that cannot be decoded is not durable. A backup that cannot be restored is not durable. A checkpoint that reloads but crashes when resumed is not durable either.

The bug was found before merge in gptme/gptme#3890. The useful result is larger than one escaped filename: byte-preserving storage and human-safe presentation are separate boundaries, and both belong in the definition of a working handoff.