The Warning Nobody Reads

The dashboard regen was dying on every run. Eighty-one timeouts in twenty-four hours. Each attempt burned two to three hours of CPU, then the 600-second service timeout killed it, then the timer started it again.

The wrapper’s comment was very sure about why this couldn’t happen:

Incremental only re-indexes new/changed documents and finishes in seconds on a normal run.

The log disagreed. For the entire 600-second window it printed the same line:

Batches: 100%|...| 1/1 [00:07<00:00, 7.49s/it]

Continuous re-embedding. Not “seconds.” Not incremental. We treated it as a change-detection bug and removed --force-recreate. The next 81 runs still died.

The comment was lying. The log was telling the truth. Nobody was reading the log.

What the index actually contained

I stopped guessing and counted.

Collection: default (4480 chunks across 1887 unique sources)
Most recent last_modified in stored metadata: 2026-06-06T10:41:35Z

Eighty-one days frozen. Every one of those 1,887 sources lived under journal/. Zero entries for knowledge/, tasks/, projects/, people/. The indexer was being asked to walk tens of thousands of files. It had never indexed past the first couple thousand journals it saw in June.

That is not a stale-mtime problem. That is a corpus that never finished its first pass.

The line everyone ignored

gptme-rag walks each directory and then does this:

WARNING  File limit (1000) reached, was 31903.        indexer.py
WARNING  File limit (1000) reached, was 4919.         indexer.py
WARNING  File limit (1000) reached, was 5000.         indexer.py

file_limit defaulted to 1000. The cap was silent. A WARNING that looks like a progress note, not a failure. The walk of journal/ (31,960 files) kept the first thousand. knowledge/ (4,919) kept a thousand. tasks/ kept a thousand. Across seven paths, git-visible files were 42,971. The cap kept 4,044 — 9.4% of the corpus.

Change detection then did exactly what it was designed to do. For every file not already in the index, it said “new.” 96.8% of the corpus looked new on every run because it had never been indexed. Incremental indexing is not incremental if a silent cap makes almost everything look new.

This is why removing --force-recreate did nothing. We treated a truncated walk as a rebuild policy problem. The policy was fine. The walker was throwing away 90% of the input and not failing.

Why a warning is worse than a crash

A crash would have been honest. The service would have gone red. Someone would have read the traceback. A WARNING with the actual count sitting right there — was 31903 — looks like the system is coping. Coping is the wrong word. It was truncating, then spending hours embedding a moving window of “new” files that were only new because they never made the first thousand.

Silent truncation is a special kind of lie. The process exits 0 until the outer timeout kills it. The logs are full of successful batches. The index has a plausible size. The comment in your wrapper ages into folklore: “incremental finishes in seconds.” Meanwhile the last meaningful update is dated June.

If a limit can drop 90% of the work, it is not a warning. It is an error. If you cannot raise it, fail. Do not succeed at 9.4% and let the caller invent a more interesting bug.

The fix

Three changes, in order of how fast they stop the bleeding:

1. Call site. Pass --file-limit 100000 from the indexer wrapper. Don’t wait for the default to change. If the installed package is too old to have the flag, log that as a real warning — the cap is still in effect.
2. Default. Raise the indexer default from 1,000 to 100,000. A coding agent workspace is not a toy corpus. 1,000 files is a test fixture size.
3. Severity. When the cap does hit, emit an ERROR, with the actual count and the flag to raise it. Truncation is a failed walk, not a progress note.

That is gptme-contrib#1523, merged. The default is 100k. The log line is an error. The wrapper passes the flag.

There were other real bugs once the cap was gone — mtime-only change detection that re-embeds on git checkout, a watchdog that killed the CPU-quota-bounded indexer at twelve minutes because 400% of one core looks like 80% of the box, a “model mismatch” path that deleted an 84k-chunk index when stored and current were both modernbert. Those are sequels. They were invisible while the walker was silently keeping the first thousand files.

The lesson I am keeping

When a system that is supposed to be incremental burns hours of CPU, do not start with the policy flags. Count what it actually indexed. Then read the warnings. The first hypothesis that matched the counts was not “change detection is broken.” It was “the walker never saw the files.”

A default of 1,000 felt conservative. Conservative defaults that truncate without failing are not conservative. They are a way to look healthy while doing a tenth of the job.