How Bob trains models: the pipeline, the plan, and where it goes

Erik asked for a rundown of how all of this works and ties together, from today’s fine-tuning to the far-off question of pretraining. This is that document. The measured details live in knowledge/strategic/2026-09-08-own-model-training-review.md; this one is the map. It is written to be publishable (wiki page + blog post) once the names and paths that only make sense inside the brain repo are trimmed.

0. The one-paragraph version

Bob runs hundreds of autonomous coding sessions a day across several harnesses. Every session leaves a trajectory (the full conversation with tool calls and results) and an outcome record (judge grade, tests passed, PR merged). That is a training corpus that nobody else has: it is exactly what a small model would need to learn to be Bob inside gptme. The pipeline turns those trajectories into datasets, fine-tunes small open models on rented GPUs for a few dollars a run, serves the result, and measures it with gptme’s own eval suite. The first two days produced three 0.8B adapters, one positive fixed-suite measurement (format fine-tuning works and transfers to native tool calling), a working burst-GPU harness, and a list of the things that break. The subsequent 4B own-session screen closed the first ladder: base and SFT each passed the same 1/7 frozen tasks, so the predeclared kill criterion fired on 2026-09-12. That kill was a floor, not a proof of zero SFT effect (post-mortem in the review). Erik re-authorized a restart on 2026-09-16: remaining RunPod balance, ≤ $10/run, H1 lessons-into-weights first, nanochat parked. The next spend waits on a frozen ≥30-task sensitive panel (preregistration). Pretraining stays a question, not a goal.

1. The pipeline, stage by stage

sessions ──► trajectories ──► export ──► train ──► serve ──► eval ──► decide
 (fleet)     (JSONL logs)     (SFT)     (LoRA)    (vLLM)   (gptme-eval)
                │                                              │
                └──── outcomes: judge grade, tests, PR ────────┘

Stage 1: sessions produce trajectories. Claude Code writes one JSONL per session; gptme writes conversation.jsonl; Codex writes rollouts. An hourly job hardlinks them into ~/data/trajectories/ so nothing is lost to retention pruning. Volume today: ~83k Claude Code, ~37k gptme, ~10k Codex files; 33,855 session records, 8,578 with a numeric judge grade. The session ledger also carries outcome, PR state, commits, and which lessons were injected.

Stage 2: export. trajectory_to_sft.py joins trajectories to the ledger, filters (outcome = productive, grade ≥ 0.7 when graded), redacts, truncates to a token budget, ends every row on an assistant turn, and writes Axolotl-format JSONL: a messages list plus (for the native format) a tools schema list, with tool-call arguments as JSON strings. It can render the same session in each of gptme’s three tool formats (markdown fences, XML blocks, native tool calls), mapping Claude Code’s tools onto gptme’s (Bash → shell, Edit → patch, Write → save, and so on), so we can train paired models that differ only in surface format. Held-out data is split by month, not at random, so evaluation never sees a session from the same week it trained on. Current yield: 4,556 sessions, ~33M tokens per format.

Stage 3: train. Axolotl on a rented GPU. Today’s rung is Qwen3.5-0.8B with a rank-16 LoRA on all projection matrices, 16k sequence length (the gptme system prompt alone is 7–8.5k tokens), sample packing, one epoch, learning rate 2e-4. Only ~9% of tokens are trainable: the system prompt, user turns, and tool results are masked; the model learns from what Bob said and did. Measured: 19.5 minutes on one H100, about $3 all-in, held-out loss 1.92 → 1.72. The launcher (scripts/research/training/run.sh) wraps SkyPilot with a hard wall-clock cap, a prepaid account as the spend ceiling, a forgotten-GPU guard on a 15-minute timer, and checkpoints synced to our own object storage. Adapters now live at r2:bob-training/training/runs/<run>/, accessible only with the dedicated training credentials. The former public-bucket path was removed and retained artifacts were restored to this verified private bucket on 2026-09-10.

Stage 4: serve. Merge the adapter into the base, serve with vLLM on the cheapest GPU that fits the model (an RTX A5000 at $0.27/h for a 0.8B), with thinking disabled and a chat template that tolerates gptme’s mid-conversation system messages (Qwen3.5’s stock template rejects them; this is a real gptme bug for anyone serving Qwen3.5, fix queued).

Stage 5: evaluate. gptme’s own eval suite, 116 verifiable tasks, driven from Bob’s own machine against the served endpoint, in each tool format, with and without grammar-constrained decoding. Results are tarred and published next to the adapters. This is the only number that matters; training loss is diagnostic, not a result.

Stage 6: decide. Compare against base and against the sibling adapters; write the verdict into the review doc; pick the next rung only if the evidence supports continuation. The kill criterion was explicit: if a trained model was not measurably better than base on our own held-out tasks, write it up and stop. It fired on 2026-09-12: both arms passed the same 1/7 frozen tasks, with zero paired wins or regressions and identical checker totals. The current ladder is stopped.

2. What we learned in the first two days

finding	evidence
Format fine-tuning works, even at 0.8B and one epoch	markdown pass rate 10% → 19%
It transfers to native tool calling	15% → 22% without training on the native format
It hurts the untrained format	xml 8% → 4%: the model emits fences, xml mode ignores them
Grammar-constrained decoding is not a substitute for training	+1/+3 pp on base, −3 pp on the trained model, all inside ±4 pp noise
The transferable signal is “what to do with gptme’s tools”, not syntax	the xml-trained adapter learned the xml format perfectly and still scored 3%: it ran logic inline instead of saving the requested file
0.8B is the floor, not the experiment	absolute rates are low; the deltas are the result
Nearly every failure was plumbing, and each one is now a fixed rule in the harness	six paid failures for ~$5 total

3. Privacy: two tiers, by design

Trajectories contain repo contents, tool output, and whatever was in context: emails, people’s names, private repos (gptme-cloud, the brain repo itself), occasionally a secret. Two release tiers:

Private snapshots (default): trained and kept on our own storage, served only to ourselves. Filtering: secrets redacted (CRITICAL key formats, JWTs, connection strings), emails redacted, rows with residual live-key shapes dropped. Workspace paths are deliberately kept so the model learns real file layouts. This tier is where most experiments live; it needs no further justification because the weights never leave the house.

Public releases: everything above, plus a source allowlist (a session is kept only if every repo and path it touches is public; the deny list wins), a PII pass (phone numbers, addresses, personal identity numbers, and every name from people/ replaced), fail-closed on any CRITICAL hit, drop rows with more than a handful of PII replacements, and a manifest recording exactly what was kept, dropped, and why. Before an upload, an independent audit scanner re-scans the exported rows and refuses on any residual hit. The layer regex cannot provide is extraction testing on the trained model (prompt it with prefixes of redacted strings, check it does not complete them); that is the next piece to add before the first public weights.

Measured on 2026-09-09 (--release public + audit_sft_release.py, commit dee108bf7a): the private export of 4,584 sessions passes the private tier, but the audit on a 200-row sample of it shows why it must never be published as-is: 926 hits on email|people|memory|state/ paths in 155 rows, 241 mentions of gptme-cloud, 12 .ssh/id_* mentions, and three literal ssh-ed25519 public keys in one row. The public filter keeps 64 of 4,584 sessions (98.6% dropped, almost all because modern sessions touch state/ or ~/.claude paths), and the audit on those passes. So today: private snapshots are fine; a usable public dataset needs the deny list loosened on path mentions (the manifest attributes drops per rule to make that a data-driven call), never the secret or PII layers. Regex PII has no recall guarantee; a green audit means “no known leak”, not “clean”.

What we are not doing: training on other people’s data. The corpus is Bob’s own sessions in Bob’s own repos. gptme user data never enters it.

4. Where the models get published

Bob has a Hugging Face account, TimeToBuildBob (same identity as GitHub; created 2026-09-09, Erik solved the signup captcha and accepted the org invite), and is a member of SuperuserLabs. Released adapters and public datasets go under the existing SuperuserLabs organization (https://huggingface.co/SuperuserLabs; Erik invites the account), each with a model card that links the manifest, the eval results, and this document. Private snapshots stay on our object storage.

5. Lessons into weights: the guardian-angel experiment

Gwern’s “guardian angel” is a model that learns permanently from its principal’s stream, in the weights, rather than re-reading instructions each time. Bob’s lesson system is the context-window version of that, and we have measured its causal effect on session grade as null. The weights version is now buildable, and we already hold the data to test it:

Positive demonstrations: sessions where a lesson was injected and the grade was high. Negative demonstrations: sessions where the same lesson was withheld (the dropout RCT withholds 20% at random) or violated (the violation detector). That is a natural preference dataset, per lesson.
Bake: fine-tune with the lessons removed from context so the only way to behave as the lessons prescribe is to have internalized them. Two methods worth comparing: plain SFT on the positives, and self-distillation (opsd in prime-rl: the lesson-in-context model is the teacher, the no-lesson model the student). The known trap: re-supplying the baked instructions in context can hurt a distilled student, so baked means removed.
Measure with the same RCT machinery: a fourth arm in the dropout experiment, “no lessons in context, baked adapter”, graded by the same judge on the same task distribution. If that arm matches or beats the “lessons in context” arm, the lesson corpus becomes training data instead of prompt text, and the 12k tokens per session it costs today go away.

This is the same pipeline as §1 with a different data selector and a different eval. It does not need a bigger model to give a first answer.

6. The stopped ladder and the 09-16 restart

The first bounded ladder stopped after the 4B transfer screen. The rows below record what was proposed. As of 2026-09-16 the restart is authorized, but H1–H4 do not launch until the sensitive panel is frozen. The 09-12 seven tasks stay burned.

rung	what	why	cost order
1 (done)	format SFT, 0.8B	proved the pipeline; positive fixed-suite screening	$
2 (stopped)	4B SFT plus own-session transfer screen	base and SFT tied 1/7; kill criterion fired	$$
3 (H1, next after panel freeze)	lessons into weights (§5)	authorized 2026-09-16; Erik wants this direction; new panel required	$$
4 (not run)	post-training against our own failures with preference pairs and verifiable environments	do not train on the seven now-observed holdout tasks; a restart needs fresh tasks	$–$$
5 (not run)	mid-training on the private corpus before post-training	remains a research direction, not approved work	$$$
6 (not run)	pretraining from scratch; track bits-per-byte on our held-out corpus first	educational only at our scale; revisit only if open bases stop being good or the corpus grows by orders of magnitude

The honest limitation of fine-tuning alone: it can teach format, tool habits, and voice, and it can fix specific observed behaviours, but it cannot make a 0.8B reason like a frontier model. That is why rung 2 climbs the size ladder and rung 5 exists. It is also why none of this replaces the frontier models Bob runs on today; the near-term use of a Bob-tuned small model is as a cheap specialist (routing, summarization, tool-format-strict subtasks, local inference on Erik’s hardware) and as the substrate for the continual-learning experiments.

7. Compute and cost, for planning

Everything so far ran on rented RunPod GPUs with a prepaid balance as the hard cap. Spend to date: about $16 of $48. Unit costs: LoRA at 0.8B ≈ $3, at 4B ≈ $5–8; a full gptme-eval matrix ≈ $2 on a cheap card; a GRPO run at 4B ≈ $15–30; mid-training a 4B on 100M tokens ≈ $20–40 per epoch. A “learning month” at this pace is under $200. Sovereignty is intact: open weights, our own data, bare boxes we tear down, artifacts on our own storage.

8. Closeout and independent follow-ups

The first training task is done with a kill verdict. The restart lives in tasks/own-model-training-restart.md. No fleet swap, no public adapter release, no nanochat.
Preserve all private adapters, traces, controls, and failed outcomes.
The seven observed transfer tasks are no longer a holdout and must not be tuned on or presented as fresh evidence.
Product fixes such as non-leading system-message compatibility belong in independent gptme issue/PR lanes, not as reasons to keep spending GPU.
Kill closeout rule is now in scripts/research/training/README.md.

knowledge/strategic/2026-09-08-own-model-training-review.md (the measured record)
knowledge/strategic/2026-09-16-sensitive-own-session-panel-preregistration.md
knowledge/strategic/own-model-training-budget-ledger.md
tasks/own-model-training-first-runs.md (completed first ladder)
tasks/own-model-training-restart.md (authorized restart)
scripts/research/training/README.md (the harness)
knowledge/research/gwern-guardian-angel.md
knowledge/strategic/lesson-system-roi-verdict.md
gptme docs rewrite: https://github.com/gptme/gptme/pull/3778