Best LLM for Summarization in 2026: Four Models, Zero Hallucinations, 4.6x Cost Spread

Summarization is the hardest common task to benchmark honestly, because "good summary" resists automated grading. So we tested the two properties that are checkable and that actually break production systems: format compliance and refusal to invent facts.

Four models, one source text, three trials per task.

The tasks

1. Strict format + no invented numbers. Summarize a board update in exactly three bullets, each under fifteen words, inventing no number not present in the source. All three conditions are machine-checkable — bullet count, word count, and every numeral cross-referenced against the source.

2. Admit what isn't there. The source states revenue, headcount, churn, runway and a raise target — but never profit. We asked for a one-sentence summary with an instruction to say so explicitly if no profit figure is stated, rather than estimating one. A pass requires acknowledging the absence. Silently inferring a profit number is the classic summarization failure and the one that does real damage.

Results

Twenty-four summaries, zero hallucinated figures, zero format violations. Every model correctly refused to invent a profit number.

The only difference is cost, and it's a 4.6× spread for identical results. Gemini 3.6 Flash produced correct, compliant summaries for $0.0006 each — about sixteen thousand summaries per $10.

What this measures, and what it doesn't

It measures the failure modes that break pipelines. A summary that quietly invents a profit figure corrupts a report. A summary that returns four bullets when your template expects three breaks rendering. Both are checkable, and all four models handled both.

It does not measure whether the summary is good. Salience — did it pick the right three facts? — is genuinely subjective, and we're not going to launder a judgment call through an LLM judge and call it a benchmark. If two models both produce factual, compliant summaries, choosing between them on quality requires reading them, with your own source material and your own idea of what matters.

Our honest read is that at this level of task, all four were fine, and the differences we could see were stylistic rather than better or worse.

The practical answer

Use Gemini 3.6 Flash for volume summarization. Perfect scores at a fifth of the next-cheapest option. At a million summaries a month that's roughly $600 against $2,700 for Sonnet 5 — same measured accuracy.

Note this is the opposite of our code-review recommendation. There, Grok 4.5 was cheapest; here it costs 2.8× Gemini. Summarization is input-heavy — a long document in, a short summary out — so the input rate dominates and Gemini's cheaper input wins. Code review is output-heavy and the ranking flips. Cheapest depends on the shape of your task, not on a blended price.

Always assert on format. Bullet counts and word limits are free to check and catch template breakage before it reaches a user.

Test the absence case explicitly. The "say so if it isn't stated" instruction worked on all four here, but it's the assertion worth keeping in your suite, because a fabricated number passes every schema check you have.

A note on our own grading

Our first pass scored GPT-5.6 Luna 5/6. It wasn't. Luna had written "$900,000", and our number extractor tokenised that as 900 and 000 — then flagged 000 as a figure not present in the source.

Luna's summary was correct. Our regex didn't handle thousands separators.

That's the third grading bug we've caught in this series, and all three shared a shape: the grader was stricter than the task, and the model was penalised for a formatting choice rather than an error. We fixed the parser to strip separators before tokenising and re-graded everything.

We keep writing these notes because a benchmark is only as trustworthy as its grader, and graders are code, and code has bugs. If a comparison doesn't tell you how it graded, you can't tell whether you're reading a result or an artifact.

Try it yourself

• The format + no-invention task — count the bullets, check every number

To compare summarization cost and compliance across models on your own documents, that's Promptster.

Run 2026-08-11 via the Promptster API, max_tokens: 4000, three trials per model per task. Grading is script-based: bullet count and per-bullet word count are exact, and every numeral in the output is cross-referenced against the source after stripping thousands separators. Costs are per summary, from list pricing against actual token usage. This tests factual restraint and format compliance only — not salience, tone, or usefulness, none of which we can grade without judgment.