Which AI Model Should I Use? A Task-Type Decision Framework Backed by Real Data

"Which model should I use?" is the most common question we hear, and it's almost always answered badly. The two failure modes are symmetrical: "always use the frontier model because it's safer" wastes 10-40x on cost; "always use the cheapest model because it's fast" ships silent bugs.

The honest answer is that the right model depends on the shape of the task. Not the domain — coding vs writing vs extraction — but the underlying cognitive pattern. Here's a framework that maps task types to tier choices, backed by five tests we ran across budget, frontier, and reasoning-tier models.

The Framework

Before the data, here's the rubric. We classify prompts along two axes:

• Complexity: does the task require multi-step reasoning, or is it a well-formed transformation?
• Ambiguity: is there one correct answer, or does quality vary on a spectrum?

The data below confirms this. Let's walk through five task archetypes.

Archetype 1: Factual Recall (Low Complexity, Low Ambiguity)

What it looks like: "When was X released?", "What are the features of Y?", "What does the Z specification say?"

What we tested: "Name 5 features added to Python 3.12 with PEP numbers" across 11 providers.

What happened:

• Perplexity (which has web search) scored 5/5
• DeepSeek / Mistral / xAI grok-3 scored 4/5
• Most cheap-tier models scored 0–2/5
• Six of eleven models confidently cited PEP 657 as Python 3.12 — it's actually 3.11

Verdict: This looks like budget work and isn't. Factual recall exposes training-data errors that are invisible on easier prompts. Either use a web-connected model (Perplexity, Google with grounding) or pay mid-to-frontier tier, or provide the source of truth in the prompt. The cheapest tier is dangerous here because its errors sound authoritative.

Archetype 2: Well-Specified Code Generation (Low Complexity, Low Ambiguity)

What it looks like: "Write a function that validates an IPv4 address with these rules." Requirements are explicit, correctness is checkable.

What we tested: IPv4 validator with 6 explicit requirements, graded on 8 providers. Full data.

What happened: All 8 produced functionally correct code. The most expensive answer ($0.003813 on Claude Sonnet 4.5) and the cheapest ($0.00 on Cerebras free tier) were equivalent in output.

Verdict: Budget tier wins outright. If you're paying frontier rates for boilerplate validation, data shaping, or CRUD generation, you're on the wrong tier. GPT-4o-mini, Gemini 2.5 Flash Lite, and DeepSeek Chat all ship production-grade code on these prompts.

Archetype 3: Structured Data Extraction (Low Complexity, Low Ambiguity)

What it looks like: "Pull these four fields from this paragraph and return JSON."

What we tested: Extracting company name, product name, launch date, and three customer types from a press release, across Gemini 2.5 Flash Lite, Claude Sonnet 4.5, and GPT-4o-mini.

What happened:

Identical factual content. 36x cost spread. And the cheapest model actually followed the "JSON only" instruction better than the frontier model.

Verdict: Budget tier. Unambiguously. Extraction is the single most over-paid task we see teams run on frontier models. If your prompt has a schema, use a budget model.

Archetype 4: Multi-Step Arithmetic (High Complexity, Low Ambiguity)

What it looks like: "A store offers different discounts on different days. Given X conditions, compute the final price." Many small steps, one correct answer.

What we tested: Tuesday 15% discount, $50 store credit pre-tax, 8.25% tax on 480 initial. Correct answer: $387.54.

What happened:

The reasoning-tier model was the worst outcome here — not because it got the math wrong, but because its reasoning tokens weren't piped through to the response field. This is a known gotcha with reasoner-family APIs: if you're not parsing the reasoning-content field separately, you get nothing back.

Verdict: Budget tier handles it, but verify reasoning-model output parsing. For well-formed multi-step arithmetic, modern budget models have the chops. The frontier tier is not a free upgrade — Claude's labeling slip would have landed a bug in an agent pipeline that trusted the intermediate steps. The reasoning tier needs extra plumbing.

Archetype 5: Creative Writing With Constraints (High Complexity, High Ambiguity)

What it looks like: "Write a 3-sentence opening with this tone, using this specific first line, with these formal constraints."

What we tested: "The clockmaker's last customer arrived at 11:59 PM." — melancholy but not sentimental, no dialogue, no -ly adverbs, end with an image.

What happened:

Gemini 2.5 Flash Lite:

The clockmaker's last customer arrived at 11:59 PM. A thin man, cloaked against the damp night air, he stood silhouetted in the shop doorway, his face obscured by shadow. In his gloved hand, he held a small, tarnished music box.

Serviceable. Followed every rule. Imagery is competent but leans on cliché (thin man, cloaked, silhouette).

Claude Sonnet 4.5:

The clockmaker's last customer arrived at 11:59 PM. She carried a brass pocket watch in her gloved hand, its face cracked in a perfect diagonal from ten to four. Through the shop window behind her, snow fell past the street lamp in slow, fat spirals that dissolved against the dark pavement.

Clear craft advantage. Specific physical details (brass, diagonal ten-to-four, fat spirals, dark pavement). Followed every rule. Noticeably more literary.

GPT-4o-mini:

The clockmaker's last customer arrived at 11:59 PM. The door creaked as it opened, allowing a gust of cold air to swirl through the dimly lit shop, where the scent of polished wood mingled with the faint tick of countless timepieces. Outside, a single streetlight flickered, casting long shadows that danced on the cracked floorboards, swallowing the remnants of the day.

Broke the -ly adverb rule ("dimly") and slid toward sentimental with "remnants of the day." Ambitious, but non-compliant.

Verdict: Frontier tier earns its keep here. Claude Sonnet 4.5's output is the only one you'd ship to a real creative project. For casual or volume use (marketing variations, draft scaffolds), budget tier is fine. For finished or client-facing creative work, pay up.

The Matrix

Synthesizing across all five archetypes, here's the decision matrix:

How to Run This Framework in Production

You don't pick a model. You pick a router.

The teams getting the best cost-quality ratio classify prompts at ingest time and route to different tiers based on task type. Two ways to implement this:

Option 1: Static routing by prompt template. If your app has a known set of prompt types (extraction, code gen, reasoning), hard-code the tier per template. Easiest, most reliable. A single Promptster scheduled test per tier catches drift.

Option 2: Dynamic routing with a budget-tier classifier. Have a cheap model classify the prompt first, then route. Costs one extra nano-tier call (~$0.0001) per request, but handles unstructured user input. Example: GPT-5-nano classifies "is this a factual question, a code request, a creative task, or math?" and routes accordingly.

Both patterns are easy to prototype on the Promptster API — run the same prompt against multiple tiers, measure the divergence, and only route to frontier when divergence is above threshold. For a tutorial on building a router, see our upcoming "LLM router in an afternoon" post on May 3.

The Habit Change

Stop asking "which model is best?" and start asking "what shape is this task?" The task shape tells you the tier. Two or three tiers cover 95% of developer workloads.

The teams overpaying are running Claude Opus on extraction jobs and wondering why the bill is $40K/month. The teams shipping bugs are running Cerebras 8B on reasoning jobs and wondering why their agent keeps fabricating. The teams doing it right are routing per task type and measuring the outcomes against a reference set.

For the cost-side of this argument, see the 300x price spread. For the quality side, see the full 11-provider consensus study.

Tests run 2026-04-18 via the Promptster MCP server. Temperature 0.1–0.7 depending on task. Individual task results available on request; rubric is explicit so you can replicate with your own prompts.