Claude Opus 5 vs GPT-5.6 Sol: We Ran Our Own Battery Instead of Reading the Benchmarks

Anthropic shipped Claude Opus 5 on July 24. OpenAI shipped GPT-5.6 on July 9, in three tiers — Luna, Terra, and Sol, cheapest to most capable. Within days there were a dozen "Opus 5 vs GPT-5.6 Sol" posts, nearly all of them reprinting the same vendor benchmark tables: Opus 5 ahead on SWE-bench Pro, ARC-AGI-3, Frontier-Bench, leading on 9 of 12 shared benchmarks.

Those numbers are real. They also measure the hardest problems anyone could construct, which is not what most production prompts look like.

So we ran our own battery. Four tasks, each with an answer a script can check — no LLM-as-judge, no scoring rubric, no vibes. Here's what happened.

The battery

Every task grades objectively. That constraint rules out a lot of interesting prompts, but it also rules out fooling ourselves.

Both models ran through the same API with max_tokens: 4000 — enough headroom that neither gets truncated mid-answer.

Three of the four tasks can't tell them apart

Both models found both bugs in the median function — the input mutation and the even-length case. Both solved the scheduling puzzle. Both emitted JSON with the exact key set and correct values, including inferring seats: 12 from "about a dozen" and trial: false from "not looking for a trial."

If your workload looks like those three tasks — and a great deal of production LLM work does — these models are interchangeable on quality. What isn't interchangeable is the bill.

What differs is speed and cost

Same verdict on all three, consistently 1.3–1.8× the latency and 1.4–1.7× the cost. Opus 5 also spends more tokens getting there — 802 output tokens on the debug task against Sol's 408, for answers graded identically.

That gap is the whole decision for a high-volume workload. At a million debug-shaped calls, the same result costs about $20,400 on Opus 5 and about $12,500 on Sol.

The fourth task is where they actually separate

The constraint task — three sentences about the ocean, no letter "e" — is trivially checkable and genuinely hard for language models, which generate tokens rather than letters.

We ran it five times per model, because a single sample on a borderline task tells you nothing:

Sol was perfect across five runs. Opus 5 got it right once, missed by a single letter three times, and produced one run with seven.

The near-misses are the interesting part. Here's an Opus 5 failure:

A cold blue vastness fills most of our world. Sharks and squid roam its dark fathoms, hunting in murky calm. Salty air, loud gulls, and crashing surf pull many folks to sandy coasts.

That's a good paragraph. It's also wrong, twice: blue and vastness. The model held the constraint across three sentences and dropped it on two common words.

Sol's answer is shorter and blunter, and correct:

Salt surf rolls on sand. Fish dart amid coral. Moonlit swirls glow at dusk.

There's a longitudinal note here too. We ran this same lipogram test in May against the previous generation, and both models failed it. One lineage has since fixed it. The other hasn't.

What this means

Vendor benchmarks and production tasks measure different things. Opus 5 leads on 9 of 12 published benchmarks and we don't dispute that — those benchmarks probe frontier difficulty. Our three ordinary tasks don't discriminate between the models at all, because both are far past the bar. Benchmark leadership predicts performance on benchmark-hard problems, not on your extraction pipeline.

Instruction-constraint adherence is its own axis. It doesn't show up in aggregate benchmark scores, and on this axis the cheaper model wins decisively. If your product depends on hard output rules — length caps, forbidden terms, strict formats, compliance language — test that specifically. Don't infer it from a leaderboard.

Match the model to the task. For work that looks like our first three tasks, Sol delivers the same verdicts 1.3–1.8× faster at 1.4–1.7× less cost. Opus 5 earns its premium on problems genuinely at the frontier — which is exactly what its benchmark lead measures, and exactly what most production prompts aren't.

A note on method, because we nearly got this wrong

The first time we ran this battery, Opus 5 scored 1 of 4 and GPT-5.6 Sol scored 3 of 4. That's a striking headline and it flatly contradicts every published benchmark, which is precisely why we didn't publish it.

It was our bug. Claude 5 models run adaptive thinking by default and return their response as a thinking block followed by a text block. The thinking block's text is empty by default, and our parser was reading the first block. Opus 5's answers were arriving correctly and being discarded before we ever saw them — it had answered the reasoning puzzle correctly the whole time.

Two things kept it from shipping: every task graded against a known answer rather than a judgment call, and a result surprising enough that we went and read the raw API response instead of trusting our own tooling. The fix is live, and there's now a regression test that sends a prompt hard enough to trigger reasoning and asserts the answer survives parsing.

We mention it because "we ran our own benchmark" is only worth something if the harness is worth something. Any comparison that hands you a surprising result without showing you the plumbing deserves a second look — including this one.

Run it yourself

Every prompt above is reproducible, and you don't need an account or an API key:

• Run the constraint test — the one that separates them
• Run the reasoning puzzle — the answer is Wednesday

To compare both models side by side on your own prompts, that's what Promptster does.

Run 2026-08-04 against claude-opus-5 and gpt-5.6-sol via the Promptster API, max_tokens: 4000, default temperature. Costs are computed from list pricing: Opus 5 at $5/$25 per million input/output tokens, GPT-5.6 Sol at $5/$30. Latency includes network round-trip from a single region and will vary. The constraint task was run five times per model; the other three were run once each, which is enough for a pass/fail on a deterministic answer but not enough to characterise variance — treat the latency and cost figures as indicative, not precise.