60 Days of Prompt Engineering: Everything We Learned From 30 Posts and 200+ Test Runs

30 days ago we published "30 Days of Prompt Engineering" — a first pass at what systematic cross-provider testing teaches you. That post was speculative in places, grounded in a handful of ad-hoc tests.

This post is the follow-up. Another month of testing. 200+ documented test runs. 30 published benchmark and tutorial posts. This is what we actually learned — the patterns that held, the surprises that emerged, and the practical advice we'd give any team building AI-backed products in 2026.

What Held Up From the First 30 Days

The 300x price spread is real. Our deeper analysis in the cost-per-quality post confirmed: between the cheapest active nano-tier and the priciest Opus tier, you pay 300x on input tokens and up to 187x on output tokens. For many workloads, that premium buys nothing.

Multi-provider testing is the highest-leverage habit. Our early hypothesis — that running the same prompt across providers is the most useful thing you can do — held up across every test we ran. See the 11-provider consensus study for the canonical demonstration.

"Evals are the new unit tests" is more than a slogan. Teams that ship without prompt regression tests ship regressions. It's just a slower kind of regression.

What Surprised Us

Self-preference bias in LLM judges is striking. We expected a tendency; we got a 3-for-3 perfect diagonal in our judge bias audit. Every judge ranked its own provider #1. This has major implications for how eval pipelines are built.

Shared training data produces shared errors at scale. Six of eleven models cited PEP 657 as a Python 3.12 feature (it's 3.11). When the majority of your "consensus" agrees on the wrong answer, you can't tell signal from noise without external ground truth. This broke a naive assumption we had about consensus-as-verification.

Open-weight models fell for prompt injection at alarming rates. DeepSeek Chat and Groq's Llama 3.3 70B both followed a trivial injection payload in our stress test. The three frontier providers (OpenAI, Anthropic, Google) resisted. The security gap between open-weight and frontier is larger than the quality gap.

Reasoning-model APIs break naive parsers. Both DeepSeek Reasoner and OpenAI's o4-mini returned empty visible content fields in multiple tests while burning thousands of output tokens. The reasoning tokens cost breakdown documented how 99%+ of output tokens can be billed but invisible.

Models that hallucinated the most confidently charged the most. Not a rule, but a pattern: the models that refused to say "UNCERTAIN" on fabricated citations (leaderboard) were mid-priced, not cheap. Confident fabrication is not a budget-tier problem exclusively.

The Patterns We'd Stake Advice On

After 60 days, the recommendations we're most confident in:

1. Route by task shape, not by brand loyalty

The task-type decision framework remains our most useful deliverable: classify the prompt by complexity × ambiguity, route to the appropriate tier. Default everything to the frontier = you overpay. Default everything to the nano tier = you ship bugs.

2. Budget tier is the pragmatic default for CRUD

For well-specified code, extraction, classification, and reformatting, GPT-4o-mini and Gemini 2.5 Flash Lite match frontier quality at 1/20th the cost. The cheap-fast-smart triangle analysis is the single most actionable post we published.

3. Never same-provider judge and candidate

The 3-judge consensus pattern cancels self-preference bias. If you have one eval setup that's providing numbers for model-choice decisions, switching to a cross-provider judge panel is the highest-ROI change available in 2026.

4. Test your injection resistance before you ship

The prompt injection stress test takes five minutes and catches 2-of-5 production providers failing on a trivial payload. If you skip this test, you deploy a vulnerability.

5. Schedule drift detection on high-stakes prompts

Providers silently update models. Your quality drops. Scheduled drift detection at $0.012/week per prompt is the cheapest insurance available.

6. Treat system prompts as design artifacts

"Context engineering by example" showed that explicit capability boundaries, uncertainty protocols, and few-shot examples move quality 30%+. Generic "you are a helpful assistant" leaves massive improvement on the table.

7. Version control your prompts

"Shipping prompts like code" is the discipline that separates teams that ship regressions from teams that catch them before merge. PRs, diffs, reviews, and eval gates on every prompt change.

What We're Skeptical About Going Into Q3 2026

Agentic orchestration as a default. The coordination tax research shows multi-agent systems underperform single-agent setups by 39-70% on non-trivial tasks. The 2026 trend toward more-agents-more-capability is likely to correct.

Long context replacing RAG. The 1M context tax analysis and RAG vs long-context framework show that for most interactive workloads, RAG still wins on cost and latency. Long context wins for genuinely unbounded-evidence tasks.

Single-model lock-in. We've been saying this for a year. Every month, the evidence for multi-provider routing gets stronger. Teams that bet their product on one provider are one outage, one pricing change, or one quality regression away from a painful migration.

What We'd Test Next (Second 30 Days)

If we were doing this again, the open questions we'd prioritize:

1. True needle-in-haystack recall at 1M tokens across providers. The published numbers are marketing; the empirical numbers would matter.
2. Long-running agent quality over 20+ turns. Most benchmarks are single-turn. Real agents degrade over turns in ways hard to predict.
3. Real-world tool-use reliability across MCP servers. The security angle is well-covered; the reliability angle is under-studied.
4. Cross-lingual evaluation. All our tests were in English. Providers diverge more on non-English inputs; we don't have good data on which ones.
5. Local-inference SLM quality on device for production workloads. The hype and the reality still don't match.

The 60-Day Summary in Three Sentences

Cross-provider testing is the single highest-ROI engineering habit for AI-backed products in 2026. The right model for your task is almost never the model you defaulted to. And the published benchmark score on a provider's launch post tells you approximately nothing about what your production workload will cost or how it will perform.

For the full arc of the last 60 days, start with the 11-provider consensus study and work your way through whichever of our 30+ benchmark and tutorial posts match the problem you're about to solve.

Thanks for reading 60 days' worth of posts. The tests themselves, and the saved-test records, live in Promptster — if you want to re-run any of them on your own workload, the comparison view is the fastest path.