30 Days of Prompt Engineering: What We Learned Comparing Models Across 11 Providers

Thirty days ago, we set out to answer a question that most teams never get around to testing systematically: which AI model is actually best for the work you do every day?

Not which model wins on abstract benchmarks. Not which one has the best marketing. Which one produces the best results for real tasks -- coding, writing, analysis, support, creative work -- when you test them head to head with the same prompts under the same conditions.

Over the course of this series, we ran hundreds of comparisons across models from multiple providers including OpenAI, Anthropic, Google, DeepSeek, xAI, Groq, Mistral, Perplexity, Together AI, Cerebras, and Fireworks AI. We scored every response using Promptster's evaluation system. We tracked costs, latencies, and quality trends.

Here is what we learned.

The Biggest Surprises

No single model wins everything

This sounds obvious when you say it out loud, but the data made it visceral. Claude Sonnet 4.5 dominated our coding comparison for debugging and code review, but GPT-5 was faster and better at generating new code from specifications. Groq's Llama models crushed the latency benchmarks but couldn't match Claude on nuanced reasoning.

Every model we tested had at least one task category where it outperformed the competition. Every model also had at least one category where it was clearly outclassed.

Cost differences are dramatic

The cheapest model for high-volume tasks was often 5-10x less expensive than the most expensive option -- for comparable quality on simpler tasks. Over a month of testing, we found that most teams are overspending by running a frontier model for every prompt, including ones where a smaller model would produce identical results.

Our cost optimization deep dive showed that a tiered approach -- routing simple prompts to cheaper models and reserving expensive models for complex tasks -- can cut AI API costs by 40-60% without measurable quality loss.

Open-source models caught up faster than expected

When we started this series, we expected the open-source vs. closed-source comparison to show a clear tier gap. It didn't. DeepSeek V3 scored within 0.2 points of Claude and GPT-5 on coding tasks. Mistral Large was nearly indistinguishable from proprietary models on creative writing. The quality gap is still real for complex reasoning, but it's narrower than most people assume.

Consensus analysis is genuinely useful

We were skeptical about consensus analysis before testing it. The idea of running the same prompt through multiple models and looking for agreement sounded expensive and slow. But it turned out to be one of the most reliable techniques for catching hallucinations and improving output quality.

When three out of four models agree on an answer and one disagrees, the disagreeing model is almost always the one that hallucinated. We found this pattern consistently across hallucination detection tests and factual accuracy checks.

Five Lessons That Changed How We Work

1. Test with your actual prompts, not generic benchmarks

Published benchmarks are useful for general guidance, but they don't predict how a model will handle your specific tasks. A model that scores well on HumanEval might struggle with your TypeScript codebase's patterns. The only benchmark that matters is the one run against your real work.

2. Temperature and system prompt matter more than model choice

We found cases where changing the temperature from 0.7 to 0.3 improved coding output quality more than switching from one frontier model to another. A well-crafted system prompt with clear constraints and examples consistently outperformed a vague prompt on a "better" model. Prompt engineering basics still beat model shopping.

3. Speed and quality need separate evaluation

A response that arrives in 0.5 seconds feels magically fast, but if it's wrong, the speed is worthless. Conversely, a perfect response that takes 8 seconds might be unusable in a real-time application. We learned to always measure both and make explicit tradeoff decisions rather than optimizing for one dimension.

4. Testing beats guessing, every single time

Before this series, we had assumptions about which models were "best" based on vibes, blog posts, and Twitter discourse. Systematic testing overturned several of those assumptions. The hallucination reduction techniques that actually worked were not the ones we expected, and the models we assumed were the most reliable sometimes weren't.

5. Your optimal model mix will change

Models get updated. Pricing changes. New providers emerge. The best model for your use case in March might not be the best in June. Building a habit of periodic re-evaluation -- even a monthly check -- keeps you from locking into a suboptimal setup.

Recommendations by Use Case

After thirty days of data, here are our practical recommendations:

These are starting points, not universal truths. Your results will vary based on your specific prompts, data, and quality requirements.

What We Would Do Differently

If we were starting this series over, we would have set up scheduled tests from day one. Promptster's scheduling feature lets you run the same prompts on a recurring basis and get alerted when quality drops. We added monitoring midway through and immediately caught a model performance regression that we would have otherwise missed.

We also would have been more disciplined about saving and tagging every test. The ability to go back and compare results across the full thirty days was invaluable for writing this recap, and the tests we didn't save are the ones we wished we had.

The Bottom Line

The single most important takeaway from thirty days of testing is this: the AI model you default to is probably not the best one for every task you use it for. Most teams pick one provider, use it for everything, and never question the choice. That default costs you money, quality, or both.

Testing takes minutes. The data changes how you build. And once you see the differences for yourself, you can't unsee them.

If you've followed along with this series, thank you. If you're just finding it now, start with the post that matches your biggest question and work from there. And if you haven't tried running a comparison yourself yet, open Promptster and test one prompt across three providers. It takes thirty seconds, and it might change which model you reach for tomorrow.

Start your first comparison now -- three free tests in sandbox mode, no API keys required.