You write "Reply with only: Nov <day>,

In our latest battery this happened on two Claude models and nowhere else. Claude Sonnet 5.5 explained its answer on 5 of 5 "reply with only" calls, and once broke strict JSON the same way. Claude Haiku 5.5 appended its own word-count check after the answer on 3 of 5 tagline calls. Fable 5.1, Opus 5.5 and Opus 5 did it 0 times, and so did all 25 configurations from other providers. Every one of the nine explained replies reported zero thinking tokens. The fix is to give the model somewhere to put the working and to make your parser tolerate it, both below.

What we ran

This comes from the four-task battery in our harder benchmark. Three of the tasks tell the model to return nothing but the answer:

• json-order: "Return ONLY a JSON object (no code fences, no other text)…"
• seven-word-taglines: "…Output only the three lines."
• launch-notes: "…Reply with only: Nov <day>,

  Ship AI with confidence. Monitor it forever.

  Find the right model before you build. Get alerted the moment quality drifts after you ship. Works where you already work — inside Claude, Cursor, and Windsurf via MCP.

  Confidence at every stage

  Whether you're choosing a model or running one in production, Promptster keeps you covered.

  Supported Providers

  • OpenAI
  • Anthropic
  • Google AI
  • DeepSeek
  • xAI (Grok)
  • Groq
  • Mistral AI
  • Perplexity
  • Together AI
  • Cerebras
  • Fireworks AI

  Works inside your AI coding assistant

  Promptster is available as an MCP server — connect it to Claude, Cursor, or Windsurf and run tests, compare providers, and check quality without leaving your editor. Published on the official MCP Registry.

  Pricing that scales with your AI usage

  Free — $0

  • 2,000 API calls/month
  • All supported AI providers
  • 50 saved test histories
  • 5 req/min rate limit

  Builder — $9/month

  • Everything in Free, plus:
  • 5,000 API calls/month
  • Compare 3+ providers at once
  • Save 500 test histories
  • CSV & data export
  • Performance drift analytics
  • Overages at $0.005/call

  Scale — $29/month

  • Everything in Builder, plus:
  • 25,000 API calls/month
  • Save 2,000 test histories
  • Priority support
  • Overages at $0.003/call

  Frequently Asked Questions

  What is Promptster?

  Promptster is an AI testing and production monitoring platform for developers building AI-powered features. Before you ship, compare quality, speed, and cost across 11 providers simultaneously. After you ship, set up scheduled monitoring to get alerted when your AI's output quality, response time, or costs drift from your baseline. Promptster also works natively inside AI coding assistants — Claude, Cursor, and Windsurf — via MCP.

  Will I actually save money with Promptster?

  Yes. Most users discover that 40-70% of their AI tasks can be handled by a cheaper model with comparable quality. A single optimized comparison that redirects even a fraction of your API spend to a cheaper provider pays for the subscription many times over.

  Do I need to provide my own API keys?

  Yes — and that's a feature, not a limitation. With BYOK, you pay providers directly at their published rates with zero markup from Promptster. There's no token surcharge, no hidden fees, and no vendor lock-in. Your keys are encrypted with AES-256 and never leave your device in plaintext.

  Which AI providers are supported?

  Promptster supports multiple providers including OpenAI, Anthropic, Google AI, DeepSeek, xAI, Groq, Mistral, Perplexity, Together AI, Cerebras, and Fireworks AI.

  How does cost tracking work?

  Every test shows real-time cost calculated from each provider's published per-token pricing, broken down by input and output tokens. After a comparison, Promptster highlights cost-saving opportunities — showing exactly how much you'd save by switching, along with quality and speed trade-offs.

  Is there a Public API?

  Yes. Promptster offers a Public API (v1) for programmatic access.

  How does production monitoring work?

  Set up scheduled tests to automatically run your prompts against any provider on a recurring schedule. Promptster tracks response time, cost, and quality over time, and alerts you if performance degrades beyond your SLA thresholds.

  How are my API keys secured?

  API keys are encrypted client-side using AES-GCM-256 before being stored.

  What plans are available?

  Free includes all supported providers, 2-way comparisons, 2,000 API calls/month, and 50 saved tests. Builder ($9/month) adds 5,000 calls/month, 3+ provider comparisons, 500 saved tests, CSV export, and data export. Scale ($29/month) adds 25,000 calls/month, 2,000 saved tests, and priority support.

  How much does a typical comparison cost in provider API tokens?

  Most comparisons cost $0.001-$0.05 in provider fees. A short prompt across 3 mid-tier models typically costs under $0.01. Promptster shows exact cost per response in real time.

  Ship with confidence. Monitor forever.

  Find the right model before you build. Know the moment something drifts after you ship. Free forever — no credit card.

  API Documentation

  lt;amount, no commas>"

n=5 per task, script graders (no LLM judge), max_tokens 4000, run 2026-10-23 through Promptster's own provider handlers. The instruction was in the user message, with no system prompt. Each task has a diagnostic that checks the format separately from the value, so "right answer plus extra text" is visible. The fourth task, stock-floor, asks for working and is the control.

The counts

Responses with anything besides the requested answer, on the three "only" tasks:

Cerebras (HTTP 402, unfunded key) and Mistral (HTTP 429) were blocked and are not in the table. Some non-Anthropic configurations did fail these tasks, but every one of those failures was a bare wrong answer: a wrong JSON total, a wrong budget, a line with the wrong number of words. None added commentary.

What the responses look like

Sonnet 5.5 on launch-notes, run 1 (one sentence cut, marked …):

Launch: Nov 21 (Thursday move; Friday's Nov 26 proposal was rejected).

Budget: $40,000 at the start, cut by $5,000 on Wednesday to $35,000. On Thursday half of that cut ($2,500) is restored, giving $37,500. …

Nov 21, $37500

The answer was right in all five runs, and in all five the last line was exactly the requested answer. The working came first.

Sonnet 5.5 on json-order, run 3: correct JSON on the first line, the calculation, and then:

Wait, the response must be only JSON, so ignore the explanation above.

It noticed the rule after breaking it. JSON.parse on the whole response throws. The other four runs were the bare object.

Haiku 5.5 on the taglines went the other way: answer first, check after. Run 1 printed three lines, then "Let me check the counts:", a corrected set, a second check and a third set. Its own counting was off. It wrote:

"refill it again and again today" = 7.

That line has six words. All five Haiku tagline runs failed the word count, which is a separate problem; here the point is that three of them came with a visible self-review attached.

The pattern: no thinking, visible working

Claude 5-family models decide for themselves whether to think before answering, and report it in usage.output_tokens_details.thinking_tokens (how that block looks in a response). Our handler does not change that setting. Here is how the explained replies line up with it, across the three "only" tasks and all five Claude models:

All nine explained replies reported 0 thinking tokens: Sonnet on json-order and launch-notes, where it never thought, and Haiku on the taglines, where it never thought. Where the same two models did think (Sonnet on the taglines, Haiku on json-order and launch-notes), they returned the bare answer every time. Fable 5.1 also skipped thinking on json-order and launch-notes and still returned the bare answer 10 of 10.

That is a correlation in 75 calls, not a mechanism. A reading consistent with it: when the model does not think first, Sonnet and Haiku sometimes do the working in the reply instead. It is not a cost problem (Sonnet's explained replies were 123 to 165 output tokens), but it is a parsing problem.

The fix, part 1: give the working a place

The control task suggests what to ask for. stock-floor says "Show brief working, then end with a final line of the form Answer: <number>". Sonnet 5.5 and Haiku 5.5 followed that format 5 of 5 times each. And in its five explained launch-notes replies, Sonnet's last line was always the exact answer. So instead of forbidding the working, ask for it in a known place:

Work it out briefly if you need to. Then end with one final line, exactly:
FINAL: Nov <day>, 


Ship AI with confidence. Monitor it forever.
Find the right model before you build. Get alerted the moment quality drifts after you ship. Works where you already work — inside Claude, Cursor, and Windsurf via MCP.

Confidence at every stage
Whether you're choosing a model or running one in production, Promptster keeps you covered.

Supported Providers

  
OpenAI
  
Anthropic
  
Google AI
  
DeepSeek
  
xAI (Grok)
  
Groq
  
Mistral AI
  
Perplexity
  
Together AI
  
Cerebras
  
Fireworks AI


Works inside your AI coding assistant
Promptster is available as an MCP server — connect it to Claude, Cursor, or Windsurf and run tests, compare providers, and check quality without leaving your editor. Published on the official MCP Registry.

Pricing that scales with your AI usage

Free — $0

  
2,000 API calls/month
  
All supported AI providers
  
50 saved test histories
  
5 req/min rate limit


Builder — $9/month

  
Everything in Free, plus:
  
5,000 API calls/month
  
Compare 3+ providers at once
  
Save 500 test histories
  
CSV & data export
  
Performance drift analytics
  
Overages at $0.005/call


Scale — $29/month

  
Everything in Builder, plus:
  
25,000 API calls/month
  
Save 2,000 test histories
  
Priority support
  
Overages at $0.003/call


Frequently Asked Questions

  
What is Promptster?
  
Promptster is an AI testing and production monitoring platform for developers building AI-powered features. Before you ship, compare quality, speed, and cost across 11 providers simultaneously. After you ship, set up scheduled monitoring to get alerted when your AI's output quality, response time, or costs drift from your baseline. Promptster also works natively inside AI coding assistants — Claude, Cursor, and Windsurf — via MCP.
  
Will I actually save money with Promptster?
  
Yes. Most users discover that 40-70% of their AI tasks can be handled by a cheaper model with comparable quality. A single optimized comparison that redirects even a fraction of your API spend to a cheaper provider pays for the subscription many times over.
  
Do I need to provide my own API keys?
  
Yes — and that's a feature, not a limitation. With BYOK, you pay providers directly at their published rates with zero markup from Promptster. There's no token surcharge, no hidden fees, and no vendor lock-in. Your keys are encrypted with AES-256 and never leave your device in plaintext.
  
Which AI providers are supported?
  
Promptster supports multiple providers including OpenAI, Anthropic, Google AI, DeepSeek, xAI, Groq, Mistral, Perplexity, Together AI, Cerebras, and Fireworks AI.
  
How does cost tracking work?
  
Every test shows real-time cost calculated from each provider's published per-token pricing, broken down by input and output tokens. After a comparison, Promptster highlights cost-saving opportunities — showing exactly how much you'd save by switching, along with quality and speed trade-offs.
  
Is there a Public API?
  
Yes. Promptster offers a Public API (v1) for programmatic access.
  
How does production monitoring work?
  
Set up scheduled tests to automatically run your prompts against any provider on a recurring schedule. Promptster tracks response time, cost, and quality over time, and alerts you if performance degrades beyond your SLA thresholds.
  
How are my API keys secured?
  
API keys are encrypted client-side using AES-GCM-256 before being stored.
  
What plans are available?
  
Free includes all supported providers, 2-way comparisons, 2,000 API calls/month, and 50 saved tests. Builder ($9/month) adds 5,000 calls/month, 3+ provider comparisons, 500 saved tests, CSV export, and data export. Scale ($29/month) adds 25,000 calls/month, 2,000 saved tests, and priority support.
  
How much does a typical comparison cost in provider API tokens?
  
Most comparisons cost $0.001-$0.05 in provider fees. A short prompt across 3 mid-tier models typically costs under $0.01. Promptster shows exact cost per response in real time.


Ship with confidence. Monitor forever.
Find the right model before you build. Know the moment something drifts after you ship. Free forever — no credit card.

API Documentation






lt;amount, no commas>

We did not rerun launch-notes with this wording, so treat it as the pattern the data points to, not a measured fix. If you need a pure JSON response, a provider's structured-output or JSON mode, where your stack supports it, should remove the problem at the source. We did not test that either.

The fix, part 2: a parser that tolerates it

Whatever you put in the prompt, parse defensively. Sonnet puts the answer first or last; Haiku puts a draft first and a revision later. "Take the first match" picks Haiku's draft. "Take the last line" picks Sonnet's "Wait…". So collect every candidate, keep only valid ones, and accept the result only if they agree:

// Every parseable {...} in the text, outermost first.
function jsonCandidates(text) {
  const out = [];
  for (let i = text.indexOf('{'); i !== -1; i = text.indexOf('{', i + 1)) {
    for (let j = text.indexOf('}', i); j !== -1; j = text.indexOf('}', j + 1)) {
      try { out.push(JSON.parse(text.slice(i, j + 1))); i = j; break; } catch {}
    }
  }
  return out;
}

function parseAnswer(text, isValid) {
  try {
    const v = JSON.parse(text);
    if (isValid(v)) return { value: v, clean: true };
  } catch {}
  const valid = jsonCandidates(text).filter(isValid);
  const distinct = new Set(valid.map((v) => JSON.stringify(v)));
  if (distinct.size === 1) return { value: valid[0], clean: false };
  return null; // none, or candidates that disagree: retry
}

For line answers, match whole lines only (/^FINAL: Nov (\d{1,2}), \$(\d+)$/), so a sentence like "Launch: Nov 21 (Thursday move…)" never counts. For the taglines, more than three lines that look like taglines means drafts; reject and retry rather than guessing which set is final, because Haiku's revisions were wrong too.

Log clean: false. It tells you how often a model is breaking the format, and it is the number to watch after a model upgrade.

What this is not

It is not the system prompt being ignored. The instruction was in the user message, and the model followed its substance: right date, right budget, right JSON. If you suspect the system prompt itself is being dropped, that is a different problem with a different fix.

It is also not "Claude does this". Three of the five Claude models in the battery did not, and Fable 5.1 and Opus 5.5 returned the bare answer 15 of 15 times each. If you cannot change the prompt or the parser, those are the Claude models to use for this kind of call. For how the wider field compared, see the open-weight vs closed-source benchmark.

Run this comparison yourself — no account, no API key

• Run the "reply with only" task — check whether you get one line or a paragraph

To run your own "only the answer" prompts across Claude models and others side by side, with output tokens per call, use Promptster.

Run 2026-10-23 through Promptster's provider handlers, five runs per task, max_tokens 4000, no system prompt. Claude 5+ models receive no temperature. "Extra text" is a failed format diagnostic (whole-response JSON.parse, exactly three tagline lines, or a reply of exactly Nov 21, $37500) on a response that also contained commentary; every such response was read. Thinking tokens are output_tokens_details.thinking_tokens from Anthropic's usage object. Raw responses are in content/blog/data/q4b-2026-battery/runs/.