The Discrimination Was in the Weights

A 4-bit Qwen3-4B running locally outranks haiku-4.5 at detecting hallucinations. Not by a little — PD 0.92 vs 0.755, AUC 0.855 vs 0.772, on the same 100-pair HaluEval sample we used as our judge calibration baseline in July.

The raw scores are badly miscalibrated (ECE 0.41). Temperature scaling on half the pairs and evaluation on the other half brings ECE to 0.04. Calibrated accuracy goes from 0.585 to 0.805. Confidence-gated escalation keeps the top 64% of answers at 88% accuracy and the top 24% at 98%.

Here is what surprised us: the ranking was already in the weights before we did any of that.

What we built

The experiment was motivated by TypeSafe’s Jev launch. Their claim: typed decisions read off the logit distribution instead of generated text, evaluated in parallel over a shared prefix, with calibrated probabilities. We wanted to know how much of that is already reproducible without a vendor.

The answer: nearly all of it, on M2 hardware, with a 4-bit quantized model.

scripts/research/system-one/s1.py implements three primitives — choice, noul, score — that mirror Jev’s API. The judge question is a noul call: Bob’s shipped judge wording from deceptive-content-detector.py rephrased as Yes/No. The system state is prefilled once into the MLX KV cache. Each question runs only its own short suffix against a deep copy of that cache. Questions see the state but not each other. The answer is the softmax over the option tokens’ logits — not a sample, not generated text.

For a single Yes/No field this is exactly constrained decoding with the probabilities retained.

The numbers

Escalation on calibrated Qwen3-4B probabilities:

Three things the numbers actually say

Discrimination lives in the small model; calibration does not. The 4B model says “No” almost always (mean P(Yes) = 0.001 on clean texts, 0.174 on deceptive). Its raw probability is not a confidence estimate — but its ranking is. The point is that the discrimination information was there before temperature scaling. Temperature scaling turns the ranking into calibrated probabilities. These are two separate jobs, and the small model handles the first one better than haiku.

Prompt wording dominates model size. We also ran the 1.7B model on a generic control prompt (“Is the answer given in the text factually wrong?”). AUC 0.534 — at chance. The same 1.7B with Bob’s judge wording: AUC 0.636. The question text carries most of the signal. Any System One style deployment needs the question wording under test, not just the model size.

Calibration is a post-hoc fix, not a training requirement. Two parameters — temperature T and bias b — fit on 50 pairs and evaluated on the other 50 bring ECE from 0.412 to 0.043. That is better calibration than haiku-4.5 raw (ECE 0.109). What a specialized vendor sells on top is calibration baked into training plus GPU latency. Neither is required to validate the approach.

What it costs

The 4B model at 4-bit on an M2: ~1.7 seconds per text in the current unoptimized Python loop. The 1.7B model: ~375 ms. Batching suffixes and keeping the cache resident would push those down. The comparison point for cost is the API: haiku-4.5 at ~$0.003–0.004 per 100 pairs (the session-judge judge runs thousands of pairs per day). The local model is free per call after the one-time setup.

What this does not settle

The HaluEval benchmark has a known style-leak issue — hallucinated answers are ChatGPT-written, correct answers are HotpotQA source text. Set B (40 ecological pairs sampled from real search queries, held locally) is the control. We did not run Set B for this experiment. We also did not test the Score primitive on the session-judge golden set, and we did not try Qwen3-8B.

These are on the task list. What is off the task list: signing up for a vendor, replacing the current trajectory grader, or changing any grading semantics. The vendor census verdict stands — the shape does not require one.

The research file with the full method, raw JSONL, and escalation curve: knowledge/research/2026-09-17-system-one-local-logprob-judge-experiment.md.