Cost per Quality, October 2026: 25 Configurations, 375 Passes, an 89x Spread

We ran 25 model configurations through the same three script-graded tasks, five times each. All 25 passed all 15 calls, 375 out of 375. On this battery quality is a constant, so cost per passed task is simply cost, and the spread between the cheapest and the most expensive configuration is 89×.

This is the October edition of the August cost-per-quality post. In August one task still separated models on reliability; this quarter nothing did, so the interesting columns are the ones the rate card does not show.

How we measured

Three script-graded tasks, no LLM judge: strict JSON extraction from a messy support ticket (code fences fail), a multi-step arithmetic word problem, and exactly three bullets of 12 words or fewer with nothing else. n=5 per task, temperature 0.7 where accepted, max_tokens 1500 (900 for Qwen on Groq, whose free tier caps output per minute), run 2026-10-19 through our own provider handlers, list prices from our pricing table. Every configuration passed every call, so the differences below are cost, latency and token overhead, not accuracy.

The table

Sorted by cost for all 15 calls. "Cost now" is what Promptster displays since 2026-10-22: hidden reasoning priced as output for Google and xAI, xAI's cache discount, and Perplexity's per-request fee, using the provider's own billed figure where xAI and Perplexity report one. "Displayed before" is what Promptster showed when we ran the battery. Where the two differ, the current figure is bold.

¹ Run at max_tokens 900 because Groq's on-demand tier caps this model at 1,000 output tokens per minute. No response was truncated.

Four configurations could not run: Qwen3-235B on Together (HTTP 400, dedicated endpoints only), both Cerebras models (HTTP 402, unfunded test key) and Mistral Small (HTTP 429 on the first call). Those are host and account problems, not model results. Total spend: $0.90 at current pricing ($0.76 as displayed before the fix).

Three things the table says

1. The hidden-token column reorders the middle

Gemini 3.8 Flash ranked 6th by the cost Promptster displayed before the fix and is 14th now. Gemini 3.6 Flash goes from 7th to 18th, and Sonar Pro from 12th to 19th. The Gemini Flash figures are now 4.7× and 6.2× the old display, Gemini 4 Argon's is 2.1×, and Sonar Pro's is 3.2×.

Google reports reasoning in a separate thoughtsTokenCount field, and until 2026-10-22 our Google handler did not add it to output, so the cost Promptster showed for Gemini thinking models was too low. That was a bug on our side. It is fixed: Promptster now counts reasoning tokens, cached input and per-request fees. On the bullet task Gemini 3.6 Flash's median visible output was 37 tokens against 911 reasoning tokens.

Grok moved both ways. Its reasoning tokens were left out for the same reason, but its large hidden input is mostly cached: Grok 4.7's median input on the JSON task was 1,382 tokens for a prompt OpenAI counts as 141, and a median of 1,280 of them were billed at xAI's cached rate, a quarter of the list input price. With both counted, Grok 4.5 is 1.31× its old displayed figure and Grok 4.7 is 0.93×, so on short prompts Grok is cheaper than its headline rate suggests. Sonar Pro moved the other way because Perplexity adds a flat $0.0025 request fee and search-context tokens to every call. More in why your token count is higher than your prompt.

2. The rate card is still a poor predictor

• GPT-5.4 Mini lists at well under half GPT-6 Sol's rates and cost more. It wrote 3,007 output tokens across the 15 calls against GPT-6 Sol's 1,134.
• GLM 5.3 lists below Claude Sonnet 5.5 and cost more, $0.0245 against $0.0209, on 5,060 output tokens versus 1,629.
• Kimi K3 lists below Claude Opus 5.5 and cost more on both hosts, $0.0715 against $0.0628, on about 4,200 output tokens versus 2,684.
• Claude Haiku 5.5 and GPT-6 Luna have identical list prices. Haiku cost 36% more because it wrote 2,532 output tokens to Luna's 1,890.

Verbosity, including reasoning billed as output, is the multiplier the pricing page leaves out, as it was in August.

3. Speed and cost do not line up

Qwen 3.8 27B on Groq was fastest at a 0.2 s median and sits sixth on cost. Fable 5.1, the most expensive row, was fifth fastest at 1.6 s. Gemini 4 Argon was slowest at 6.9 s for less than half Fable's cost.

What you get for 89×

Same 15 tasks, same 15 passes. Here is what separates GPT-6 Luna at $0.0011 from Claude Fable 5.1 at $0.0974.

Fable is faster. Median 1.6 s against 2.3 s; 25.0 s of wall time for all 15 calls against 35.0 s.

Fable is more repeatable. All five of its JSON answers were byte-identical, as were four of its five word-problem answers. Luna wrote the refund as 89.50 in three runs and 89.5 in two, and produced four different word-problem texts in five runs.

Fable writes plainer text. Luna used inline LaTeX such as \(48 \times 1.25 = 60\) in three of five word-problem answers; Fable never did, and laid out one line per shelf with a weight per shelf where Luna wrote a short paragraph in four of five runs.

Luna spends its tokens thinking. It wrote more output than Fable, 1,890 tokens to 1,491; on the bullet task its median was 138 output tokens, 90 of them reasoning. Fable reported zero thinking tokens on all 15 calls.

The answers are interchangeable. The bullet task, one run each:

Luna:  - Ensure reproducible outputs across deployments and experiments.
       - Avoid regressions from silent model updates.
       - Support audits, validation, and compliance requirements.

Fable: - Ensures reproducible outputs across runs and environments.
       - Prevents silent behavior changes from unannounced model updates.
       - Keeps evaluations, prompts, and tests valid against a known model.

So the 89× buys about 0.7 seconds of median latency, repeatability and plain text, none of which our grader scores. Most of it is available for less: Claude Haiku 5.5 costs 1.4× Luna, had a 1.0 s median (faster than Fable) and wrote plain text. If you need byte-identical output, parse and normalize rather than paying for it.

What this does not mean

It is not a quality ranking. Three single-turn tasks cannot measure long-horizon agentic work or hard coding, which is where the expensive rows are sold. And n=5 per task shows nothing failed; it does not show any row never fails.

Some prices are temporary. Gemini 4 Argon is on promotional pricing until 2026-11-30, then $4/$20; both Gemini Flash rows until 2026-12-31, then $1.50/$7.50. DeepSeek's own rows are peak-hour rates; off-peak is half. xAI's cache discount is applied through its reported cost. No other cache discounts are.

How to use it

Start at the top of the table and move down only when your own prompts fail. On extraction, arithmetic and formatting, the first five rows did everything the last five did. For Gemini, Grok and Sonar Pro, read the cost-now column, not the list rate. And measure tokens, not rates: four list-price comparisons above came out backwards once output was counted. For the open-weight angle, see the open-weight vs closed-source benchmark.

Try it yourself

• Run the three-bullet test — every row in the table passed it five times out of five

No account, no API key. To run your own prompts across the cheap and expensive ends of this table and see the token counts that drive the bill, that's Promptster.

Run 2026-10-19 through Promptster's own provider handlers, three tasks, five runs each, max_tokens 1500 (Groq Qwen 900), temperature 0.7 where accepted (GPT-5+ and Claude 5+ reject it and receive none). Cost now is calculateCost() with the usage fields Promptster passes since 2026-10-22 (reasoning tokens, cached input, provider-reported cost), repriced from the stored usage objects. Displayed before is the same function on input and output tokens only. Latency is median wall-clock per call from one machine, indicative only.