Cost per Quality, August 2026: What Unreliability Actually Costs You

"Cost per quality point" is a tidy idea that usually can't be computed, because you need a quality score that actually varies.

We ran fifteen model configurations through a hard battery this month. Fourteen passed every task. On that battery the metric is degenerate — quality is a constant, so cost per quality is just cost, and we already published that table.

But we have one task where reliability genuinely varies, and it produces a version of this metric that's worth your attention: what an unreliable model costs once you account for having to run it again.

The retry tax

The task is a hard output constraint — three sentences with no letter "e" — which a script verifies in one line. Either the output is usable or it isn't. Across up to twenty trials per model:

Cost per usable output is just cost-per-call divided by success rate — the expected spend to get one output you can actually ship, assuming you retry until you do.

Claude Opus 5 succeeds 47% of the time on this task. Its $0.018 call becomes a $0.038 usable output. Against Gemini 3.6 Flash's $0.000269, that's a 141× difference in the cost of getting a constraint-satisfying answer.

Why the sticker price hides this

Two models with the same per-call cost are not equivalent if one needs twice as many calls. The retry tax is invisible in every pricing comparison we've seen, including ours until we computed it.

It compounds with the verbosity effect we've measured repeatedly. Opus 5 is expensive per call because it's wordy, and expensive per usable output because it's unreliable on this task. The two multiply.

Note also the Kimi K3 rows: same model, same price, two hosts — one scored 5/5 and the other 4/5, producing a 1.00× and a 1.25× retry tax. That difference is sampling noise, not a real gap, and it shows how easily a small sample manufactures a retry tax that doesn't exist. Which brings us to the caveats.

What this does not mean

It is not a general quality ranking. This is one task measuring one property — adherence to a hard output rule. Opus 5's 47% here says nothing about its reasoning, coding, or long-horizon agentic performance, where its published benchmark position is strong and our battery cannot reach.

The sample sizes differ. Opus 5 has 19 trials and Grok 4.5 has 5. A 100% rate over 5 trials is consistent with a true rate anywhere from about 55% upwards. Treat the 100% rows as "no failures observed," not "never fails."

Retrying may not be your actual strategy. If you validate and fall back to a cheaper model, or fix the output programmatically, your real cost differs. The retry tax is the cost of the naive strategy — which is what most systems do by default.

How to use this

Compute it for the failure mode you actually care about. Pick the check your product genuinely runs — schema validation, a required format, a factual assertion — measure the pass rate over at least ten trials, and divide cost by that rate. That number, not the rate card, is what a working output costs.

Assume you'll retry, and price it in. Any pipeline with a validation step has a retry tax. If you've never measured it, you don't know your unit economics.

Don't buy reliability by moving up a tier. The most expensive model here has the worst retry tax, by a wide margin. On this axis price is not just a poor signal — it's an inverted one.

Validate rather than trusting. A one-line check on the output makes the reliability difference recoverable at any tier. Without it, a 47% success rate is silent corruption rather than a retry.

Try it yourself

• The constraint test — run it several times on one model and watch the rate emerge

To measure pass rates and cost per usable output across models on your own checks, that's Promptster.

Measured 2026-08-04 to 2026-08-11 via the Promptster API at max_tokens: 4000. Trial counts vary by model because they accumulated across several posts in this series; each row states its own n. Costs are per call from list pricing against actual token usage, and cost-per-usable-output assumes retry-until-success with independent trials. Claude Opus 5's 19 trials exclude one run that returned a provider error rather than a model failure. Success is binary and script-checked: zero instances of the forbidden letter, non-empty output.