Unknown Is Not Expensive
Mapping unknown cost to 99 made every unpriced model infinitely expensive. Missing data is not a large number. Skip the axis.
Our Pareto model selector ranked models on cost, latency, and quality. Cost tiers were integers: free=0, low=1, medium=2, high=3. Unknown was 99.
That looked tidy. It was a lie.
A model with no price could never sit on the frontier, even if it was faster and better than every known option. Incomplete data had been encoded as infinite cost. The ranking then did exactly what it was told: it treated “we don’t know” as “this is the worst thing in the catalog.”
What 99 actually did
Domination is supposed to be strict. Model A dominates B if it is at least as
good on every axis and strictly better on one. With unknown → 99, a free
model with worse latency still beat an unknown-cost peer on the cost axis
every time. The unknown never got to compete on the axes we did measure.
That is not conservative. Conservative would be: do not claim a budget fit you cannot prove. Mapping missingness to a sentinel that still participates in comparison is the opposite — it invents a ranking from a gap.
Budget filters need a number. Ranking does not.
The fix
COST_ORDER["unknown"] is now None. ModelPoint.dominates() skips the cost
axis when either side is unknown. Latency and quality still decide. Budget
filters drop unknown-cost models, because you cannot prove they fit. Sorting
and --optimize cost no longer unary-minus an int | None.
The live registry still reports 5/30 models on the current front. The six
unknown rows stay off it because they lose on latency and quality against
free low-latency peers — honest incomplete-data Pareto, not a fake-99
penalty.
Shipped in a514408a44. Tests cover unknown-vs-cheap peers, budget skip, and
knee-point sorting without unary-minus on a maybe-int.
The general rule
If a field is missing, do not pick a large integer and hope ranking “does the right thing.” Either:
- Exclude the axis from comparison, and let the remaining axes decide, or
- Exclude the row from a constraint that requires a real value (budget, SLA, a hard cap).
Those are different operations. Mixing them is how “unknown” becomes “infinitely expensive” in one code path and “free enough to try” in another.
Filling real prices in the registry is still a separate quality step. The encoding lie is gone. That is the part that was silently steering every recommendation.