We Made Our LLM Benchmark Harder. 25 of 30 Configurations Still Went 20/20. Here Is How the Other Five Failed

October's battery was a tie: every configuration that ran passed all 15 of its calls, 375 out of 375. A benchmark nothing fails can only rank on cost, so we wrote four harder tasks, each with traps, and ran them on 30 configurations. The question for this post is simple: where do current models actually fail?

25 of the 30 still went 20 for 20. The other five dropped 28 calls between them, and no two failed the same way: wrong totals inside valid JSON (Perplexity Sonar Pro), miscounted words (Claude Haiku 5.5), a host token cap (Qwen 3.8 27B on Groq), a missing SKU- prefix (gpt-oss-120b on Together) and an explanation tacked on after a JSON object (Claude Sonnet 5.5). Overall, 572 of 600 calls passed.

The four tasks

Each prompt is under 500 characters, so the link at the end of this post loads it intact.

Every task also records a diagnostic that is not scored. It separates "wrong value" from "right value, broke a format rule": JSON after stripping one code fence, a literal Answer: <n> last line, the tagline rules minus the word count, and a reply that is exactly Nov 21, $37500.

How we measured

n=5 per task per configuration, script graders (no LLM judge), max_tokens 4000 (900 for Qwen 3.8 27B on Groq, whose free tier caps output at 1,000 tokens per minute), run 2026-10-23 through Promptster's own provider handlers. Cost is cost_usd, what Promptster displays, from list prices in shared/pricing.ts; xAI and Perplexity report their own cost and we use it. Latency is median wall-clock per call from one machine. October ran at max_tokens 1500; we raised it because 1500 cut off Gemini 3.8 Flash's thinking on the arithmetic task in a pilot, which measures the cap, not the model.

Results

Sorted by spend for all 20 calls. Bold marks a task where the configuration dropped a call.

¹ Run at max_tokens 900; every stock-floor call hit that cap (see below). ² xAI's reported cost, which includes its cached-input discount. Our estimate that ignores the discount is higher: $0.0924, $0.1188 and $0.1387 for Grok 4.5, 4.6 and 4.7. ³ Perplexity's reported cost, including its per-request fee. Our sonar-pro calls went to Perplexity's chat-completions URL, which Perplexity now serves server-side as an Agent API effort: fast request running openai/gpt-6-luna. These are results for that backend, not the older Sonar model.

Blocked, not failed: both Cerebras configurations (gpt-oss-120b, qwen-3.8-27b) returned HTTP 402, payment required, on their first call; our test key is unfunded. Mistral Small returned HTTP 429 (code 1300) on the first call of each task and again after a 30-second retry. These are account problems, and we have no results for them.

On the first pass, Kimi K3 on Together timed out at 120 s on 2 of 20 calls and Qwen 3.8 27B on Groq returned 429 on 3 tagline calls. All five passed on a rerun, which is what the table shows.

How the five failed

We read every failing response. Two of the five configurations never produced a wrong value, only a format slip.

Sonar Pro got the arithmetic wrong inside valid JSON. Three json-order responses parsed, had the right SKUs and unit count, and the wrong total: 426, 414 and 294 (correct: 168). Two launch-notes answers had the right date and the wrong budget: Nov 21, $38500 and Nov 21, $38250. Its two tagline failures were word count only: one line of 8 words in each.

Claude Haiku 5.5 could not hold seven words. All five tagline runs failed. In three it printed the taglines, then a line beginning "Let me check" and a revised set, which breaks "output only the three lines". In run 4, one line was "refill it daily and carry the planet with you", nine words. Only one of the five runs passed every rule other than the count. Haiku passed the other 15 calls, including the JSON trap and the ten-step arithmetic.

Qwen 3.8 27B on Groq lost all five stock-floor calls to the 900-token cap (finish_reason: length) partway through its working. That is a host limit, not a wrong answer; the stored working matches the correct chain as far as it goes. The other losses are model errors: json-order totals of 128, 198, 198 and 150, launch-notes budgets of $42500 and $35000, and taglines of five and six words. Do not rank this model on reasoning from this run.

gpt-oss-120b on Together returned "skus":["A07","B12"] once, with the SKU- prefix gone and everything else right. Groq and Fireworks serve the same weights and went 20/20.

Claude Sonnet 5.5 returned correct JSON, then a paragraph of working, then: "Wait, the response must be only JSON, so ignore the explanation above." JSON.parse fails on that response. On launch-notes it gave the right answer every time and explained it every time, despite "Reply with only". The grader passes that (it scores extraction), but the diagnostic shows 0 of 5 Sonnet replies were the bare answer. On that task every other Claude model in the battery (Haiku 5.5, Fable 5.1, Opus 5.5 and Opus 5) and every configuration from another provider gave a bare answer, right or wrong.

What the traps did not catch

• No configuration used the tracking ref as the PO. Every parsed response had po: null.
• Nobody fenced JSON. Strict and lenient parsing gave the same result everywhere, as in October's extraction test.
• Every configuration that finished stock-floor got 587. Nobody rounded to nearest.

What it means

The cheapest row passed everything. GPT-6 Luna went 20/20 for $0.0019; Claude Fable 5.1 went 20/20 for $0.2562, 137× more. Haiku 5.5 is the second-cheapest row, and the first model at its $0.10/$0.50 rate card to lose calls on this battery or October's.

Fewer than half the failures were wrong reasoning. Of the 28 dropped calls, 5 were a token cap, 2 had the right values in the wrong form (Sonnet's trailing explanation, Together's prefix), 10 were wrong word counts and 11 were wrong numbers. Add Sonnet's five explained launch-notes answers and Haiku's three answer-then-revision tagline responses, which a strict parser would also reject: a test that checks only values misses a lot of what breaks in production.

n=5 makes a single miss a single miss. Sonnet 5.5's 4/5 and Together's 4/5 are one response each. Sonar Pro at 13/20 and Haiku's 0/5 on one task are patterns.

For the open-weight rows against closed ones, see October's open-weight comparison and the longer-running open-weight vs closed-source benchmark.

Run this comparison yourself — no account, no API key

• Run the seven-word tagline task — count the words in each line yourself

To run all four tasks on your own shortlist and see tokens and cost per call, use Promptster.

Run 2026-10-23 through Promptster's provider handlers: 33 configurations, four tasks, five runs each, max_tokens 4000 (Groq Qwen 3.8 27B at 900). Temperature 0.7 where accepted (OpenAI GPT-5+ and Claude 5+ drop it). Script graders only. Costs are cost_usd from calculateCost() in shared/pricing.ts, using xAI's and Perplexity's reported cost where present. Latency is wall-clock from one machine. Raw responses, usage objects and the first-pass log are in content/blog/data/q4b-2026-battery/.