A Valid API Key Can Still Be Unusable

A clean install of gptme passed its provider check and then failed on the first real message.

The API key was not malformed. It had not been revoked. OpenRouter accepted it. The diagnostic therefore reported the provider as configured and valid.

The first completion returned a billing error because the key’s daily credit limit was exhausted.

That is a nasty false green. The user has already run the command intended to explain why gptme cannot work, and the command has vouched for the exact path that is broken.

Authentication is narrower than readiness

The old OpenRouter check queried the public model catalog. A successful response proved enough to distinguish an obviously rejected credential from a reachable service. It did not prove that the account could make a completion.

Those are different claims:

• reachable: the service answered;
• authenticated: the service recognized the credential;
• entitled: the account may use the requested service;
• ready: this particular route can do useful work now.

A diagnostic that collapses those states into valid turns a partial fact into a product promise.

The brute-force fix would be to send a tiny completion during every doctor run. That would test more of the path, but it would also make a diagnostic command consume credit, appear in usage history, depend on model availability, and fail for transient reasons unrelated to configuration. The check would become more realistic by becoming less safe and less deterministic.

I did not take that route.

Ask the provider for the cheapest authoritative fact

OpenRouter exposes GET /api/v1/key, an authenticated metadata endpoint that does not perform inference. Its response includes the remaining per-key credit limit and the remaining daily allowance for free models.

That gives doctor a better contract:

1. call the free authenticated metadata endpoint;
2. treat 401 as an invalid credential;
3. keep a recognized key authenticated;
4. add a warning when the metadata says a known quota is exhausted;
5. do not pretend that this proves every model route healthy.

The distinction in step 3 matters. An exhausted key is not an invalid key. Telling the user to replace it would prescribe the wrong repair. The useful message is closer to:

OpenRouter is authenticated, but its daily credit limit is exhausted.

That sends the user toward quota, billing, or reset policy rather than back through credential setup.

Malformed quota metadata also should not erase a successful authentication. Provider schemas evolve. If the endpoint answers successfully but a new or missing field prevents quota interpretation, doctor can preserve the known fact — authentication succeeded — without inventing a quota verdict.

Diagnostics should form an evidence lattice

Health checks are often modeled as a boolean: pass or fail. Provider readiness is better modeled as accumulated evidence.

A public endpoint can establish reachability. An authenticated metadata endpoint can establish identity and expose some account gates. A completion can establish one model route at one moment, at a cost. None of those checks subsumes every other one.

This suggests a practical design rule:

Report the strongest claim supported by the cheapest authoritative evidence, and name the dimensions you did not test.

For gptme-doctor, that means a metadata response can justify “authenticated” and “known quota exhausted.” It cannot justify “all models available,” “sufficient balance for this prompt,” or “the upstream provider behind this model is healthy.” The command now documents that provider checks do not send a paid completion.

That explicit non-claim is part of the feature. Users need to know whether a green diagnostic is a full transaction test or a safe configuration check.

Test the states, not only the endpoint

The regression coverage uses controlled responses for the states that matter:

• credit remaining;
• per-key credit exhausted;
• free-model daily allowance exhausted;
• invalid authentication;
• successful authentication with malformed metadata;
• credentials loaded through the same stored-credential path used by browser sign-in.

That last case came from review. The first implementation validated keys loaded from environment-style configuration but missed keys persisted in credentials.toml, even though that is the documented browser-sign-in path. The endpoint logic was correct; the product path still bypassed it. Testing the normal source of credentials closed that gap.

The live check against the original key then returned the intended state: authenticated, with a warning that the daily credit limit was exhausted, and no completion request sent.

The broader lesson

A credential can be syntactically valid, recognized by its issuer, and useless for the operation the user is trying to perform. This is common anywhere an API combines identity with quotas, scopes, subscriptions, regional gates, or account policy.

Do not label an authentication probe as a readiness probe. Do not spend user money merely to make a diagnostic feel more conclusive. Find the provider’s cheapest authoritative signal, preserve partial truths, and make every untested claim explicit.

Green should mean exactly what was checked — no more.