Can LLMs Count Words? Exactly 7 Words Broke 3 of 30 Models; '12 Words or Fewer' Broke None
By Promptster Team · 2026-11-07
You ask for exactly seven words and get six, or eight, or a line that is right next to one that is not. Is the model ignoring the limit, or can it not count?
On our battery, most models can. 27 of 30 configurations wrote three seven-word taglines correctly on 5 of 5 runs, 405 lines without a miss. Three did not: Claude Haiku 5.5 (0/5), Qwen 3.8 27B on Groq (2/5) and Perplexity Sonar Pro (3/5). Two of them broke nothing but the count. In October, every configuration passed "three bullets of 12 words or fewer", including these three. An exact count is harder than a maximum, and the models that failed it were the ones that did not reason before answering.
The task
Write three taglines for a reusable drinks flask. Rules: one tagline per line;
each exactly 7 words; all lowercase; no punctuation, digits, numbering or
bullets; never use the words "water", "bottle" or "eco"; every tagline must
include the word "refill". Output only the three lines.
A script grades it: exactly three non-empty lines, each only lowercase letters and single spaces, seven words split on whitespace, no banned word (plurals included), and "refill" present. A second check, not scored, applies every rule except the word count. That separates "can't count" from "broke something else".
n=5 per configuration, no LLM judge, max_tokens 4000 (900 for Qwen 3.8 27B on Groq, whose free tier caps output per minute; no tagline response came near it), run 2026-10-23 through Promptster's own provider handlers. Cerebras (HTTP 402, unfunded key) and Mistral (HTTP 429) were blocked and have no results.
Results
| Configuration | Passed | All rules except count | Line lengths on failing runs |
|---|---|---|---|
| 27 configurations (OpenAI, Anthropic, Google, xAI, DeepSeek, Groq, Together, Fireworks) | 5/5 each | 5/5 each | — |
| Perplexity Sonar Pro¹ | 3/5 | 5/5 | 7, 7, 8 and 7, 7, 8 |
| Qwen 3.8 27B (Groq) | 2/5 | 5/5 | 6, 6, 6 · 5, 5, 5 · 7, 6, 6 |
| Claude Haiku 5.5 | 0/5 | 1/5 | see below |
¹ Our sonar-pro calls went to Perplexity's chat-completions URL, which Perplexity now serves server-side as an Agent API effort: fast request running openai/gpt-6-luna. These results describe that backend, not the older Sonar model.
Sonar Pro and Qwen broke only the count. Every one of their lines was lowercase, unpunctuated, free of banned words and contained "refill". Sonar Pro overshot by one word, once per failing run: "sip boldly refill endlessly and enjoy every journey" is eight. Qwen always undershot; run 4 was three five-word lines, starting with "refill your favorite drink daily".
How Haiku 5.5 failed
Haiku is the interesting case, because it tried to check. In three of five runs it printed three taglines, then a line beginning "Let me check", then a revised set. That alone fails "output only the three lines". But its counts were also wrong. From run 1:
"refill it again and again" = 6 words, so that fails.
That line has five words. Later in the same response it called a seven-word line eight words and a six-word line seven. Run 3 called an eight-word line nine. When it numbered the words one at a time, as in "refill(1) it(2) daily(3)…", the counts were right; when it gave a total, they were not.
The other two runs had no self-check. Run 2 passed every rule but had a six-word line. Run 4 had a nine-word line, "refill it daily and carry the planet with you", and wrote "refills" instead of "refill" in another.
Haiku passed the other 15 calls in the battery, including strict JSON and a ten-step arithmetic chain. This is a counting problem, not a general one.
The pattern: the passing models reasoned first
On this task every configuration that went 5/5 reported reasoning or thinking tokens on every run, with one exception. Some examples, per run:
| Configuration | Reasoning tokens | Visible output tokens |
|---|---|---|
| Gemini 3.6 Flash | 853–1,387 | 23–25 |
| Grok 4.7 | 413–633 | 24–26 |
| Claude Sonnet 5.5 | 192–287 | 227–323 total |
| GPT-6 Luna | 130–206 | 162–241 total |
| Gemini 4 Argon | reported on 2 of 5 runs (2, 590) | 27–28 |
The three that failed reported none. Haiku's thinking-token count was 0 on all five runs; Sonar Pro and Qwen billed 20 to 27 output tokens, which is the three lines and nothing more. Gemini 4 Argon is the exception: it reported no reasoning on three of its runs and passed all five.
That is a correlation over 150 responses, not a proof. It fits a simple account: models generate tokens, not words, and a word can be one token or several, so getting exactly seven means counting and revising. Models that think do that before you see the answer. Haiku did it in the answer, and miscounted.
Why "12 words or fewer" was easy
October's battery asked for exactly three bullets of 12 words or fewer, with nothing before or after them. (The prompt said "each under 12 words"; the grader accepts 12, and one bullet was exactly 12.) Every configuration that ran passed 5 of 5, 125 of 125 responses, including Haiku 5.5, Qwen 3.8 27B on Groq and Sonar Pro (source: content/blog/data/q4-2026-battery/summary.json; that run used max_tokens 1500).
The bullets show why. Across the 375 bullets, lengths ran from 4 to 12 words, with a median of 8. Only one bullet was 12 words and four were 11. Under a maximum, a model can aim well short and never get near the limit. Under an exact count there is no slack: six is as wrong as eight.
| Q4 three bullets (10-19) | Q4b seven-word taglines (10-23) | |
|---|---|---|
| Rule | each 12 words or fewer (graded ≤12) | each exactly 7 words |
| Configurations at 5/5 | 25 of 25 | 27 of 30 |
| Haiku 5.5 / Qwen 3.8 27B (Groq) / Sonar Pro | 5/5 / 5/5 / 5/5 | 0/5 / 2/5 / 3/5 |
What to do about it
Use a maximum if your real constraint is a maximum. Most "exactly N words" requirements are really "fits in this space". A maximum is a rule every model in October's data kept.
Count in code, never trust the model's count. Haiku's self-check was wrong more often than right. A check is one line:
const ok = lines.length === 3 &&
lines.every((l) => /^[a-z]+( [a-z]+)*$/.test(l) && l.split(' ').length === 7);
On failure, retry, or ask for more candidates than you need and keep the ones that pass. We did not measure the over-generate approach, but with 405 of 405 lines right on the passing models and 13 of 15 on Sonar Pro, a surplus of candidates is cheap insurance.
If you need exact counts from a cheap model, pick one that reasons. GPT-6 Luna, at the same list price as Haiku 5.5, went 5/5 here and reported reasoning on every run. Or enable thinking where the provider lets you; we did not test that on Haiku.
Define "word" the way your grader does. Ours splits on spaces after forbidding punctuation, which removes the hyphen and contraction arguments. If your format allows "drink-ready" or "don't", decide in advance whether that is one word or two, because the model will not know.
For the full battery this task comes from, see where models fail on our harder benchmark, and for open-weight against closed models, the open-weight vs closed-source benchmark.
Run this comparison yourself — no account, no API key
- Run the seven-word tagline task — count the words in each line yourself
To run your own length-constrained prompts across models and check them side by side, use Promptster.
Run 2026-10-23 through Promptster's provider handlers, five runs per configuration, max_tokens 4000 (Groq Qwen 3.8 27B at 900). Temperature 0.7 where accepted. Words are whitespace-separated after the grader rejects any non-letter character. Reasoning tokens are each provider's reported field (reasoning_tokens, thoughtsTokenCount, or Anthropic's thinking_tokens). October bullet lengths are counted from the stored responses in content/blog/data/q4-2026-battery/runs/; this run's are in content/blog/data/q4b-2026-battery/runs/.