120 Days of Prompt Engineering: The Models Were Fine. Our Plumbing Wasn't.

Four months ago we started publishing benchmarks instead of opinions. This series ran thirty posts across August, testing roughly twenty model configurations on identical, script-graded tasks.

Here's what held up.

1. Capability converged, and price stopped predicting it

Across a 17× price range — Grok 4.5 at $0.0033 for our battery to Claude Fable 5 at $0.0561 — every configuration passed every task. We built harder tasks specifically because an easier battery stopped separating models. The harder one didn't separate them either.

The single exception in the whole month was Mistral Large failing two of four — notable partly because it's also among the cheapest options on the board.

For ordinary, verifiable, one-shot work, the honest advice is now: start cheap and prove you need more. The burden of proof has flipped.

What this doesn't cover, and we said so in every post: long-horizon agentic work, sustained tool chains, multi-hour sessions. That's where flagship benchmark leads are earned and our single-turn battery can't reach.

2. The rate card is a bad predictor of the bill

Five separate times, list pricing pointed the wrong way:

The hidden variable is verbosity — output tokens ranged 286 to 1,751 for the same three answers. It regularly exceeds the rate difference between tiers, and it's why the cheapest model inverts by task shape: Grok wins output-heavy work, Gemini wins input-heavy work, same two models.

3. Rule-following runs inverse to price

The one axis that consistently separated models was adherence to a hard, checkable output rule — and the cheap models won.

Gemini Flash went 20/20. Claude Opus 5 went 13/29 across five sessions, at roughly ten times the cost. Priced as cost-per-usable output, Opus 5's 47% success rate turns a $0.018 call into a $0.038 result — a 2.11× retry tax that appears on no pricing page.

If your product depends on a format rule, measure the pass rate. Don't infer reliability from tier, and validate the output regardless — that works at any price and costs nothing.

4. Where you rent the model can matter more than which model

Kimi K3 runs on Together and Fireworks with identical weights at an identical price. Fireworks was 1.8–2.5× faster across two different models, with roughly half the variance.

That's larger than most model-to-model differences we measured, and it appears on no leaderboard.

5. Most published comparisons — ours included — report noise

This is the one we'd most like to see picked up.

We measured Claude Opus 5 four times on the same battery and got 26% cost variance. We ran the same constraint test five separate times and got 1/5, 3/5, 4/5, 0/5, 5/9. And we got 5/5 from one host and 4/5 from another on identical weights, which is a difference that cannot exist and is therefore a measurement of our own sampling.

We published five-trial results early in this series, then doubled to ten once the evidence said five was insufficient. Those earlier numbers are directional. So is anyone else's.

If a comparison hinges on a gap under about 25% from a single run, it hasn't measured anything yet.

6. The thing we didn't expect: the models were the reliable part

We investigated six results that looked like model failures. Five were ours.

Only one was a genuine model behaviour: Opus 5 declining to review code containing a SQL injection, three times out of three, under a cyber-safety policy.

Every one of ours was invisible to smoke tests, because smoke tests use trivial prompts and trivial prompts don't trigger reasoning, refusals, or verbose working. And every one produced a result that flattered a narrative — a clean generational-improvement story, a surprising upset, a tidy precision ranking.

That's the tell. A benchmark result that makes a better headline than you expected deserves more scrutiny than one that doesn't. We caught all five by re-reading raw output when a number looked too good.

What we'd tell someone starting this

Grade against known answers, never with an LLM judge. Every task in this series has an answer a script checks. That's what made the grader bugs findable — and what let us publish "we couldn't measure this" instead of manufacturing a ranking.

Read the raw output before you believe your own scores. Five times.

Ten trials minimum, and say your n.

Test with prompts hard enough to make the model work. 2 + 2 passes no matter how broken your integration is.

Publish the failures. The posts where our design was wrong — the precision test with no ground truth, the grounding question that ignorance answered correctly — were more useful than the ones that worked, and they're the reason to trust the ones that did.

Reproduce any of it

Every prompt in this series runs with no account and no API key:

• The interval-merge bug
• The constraint test — run it several times and watch it vary

To run any of this against your own prompts across models, with the token counts that drive the bill, that's Promptster.

This retrospective draws on runs conducted 2026-08-04 to 2026-08-11 via the Promptster API and, where a capability wasn't exposed there, directly against provider APIs. Every figure is dated and sourced in the post it came from. Costs are computed from list pricing against actual token usage; prices and model behaviour change, so re-verify before relying on any table — including ours.