My LLM's JSON Parses, but the Total Is Wrong: Why Schema Validation Misses Bad Arithmetic

Your JSON parses, your schema validator says it is fine, and the order total is wrong. That is not a parsing bug. A schema checks that total is a number; it cannot check that it is the right number.

We measured this on 30 model configurations. Every field the model copied out of the text was right in 148 or more of the 149 parsed responses. The one field it had to calculate, the total, was wrong 7 times, and each of those seven responses would pass any schema you could write for the object. The fix is to stop asking the model for the arithmetic: have it extract the line items, and compute the total in your code.

The task

From our harder four-task battery:

Return ONLY a JSON object (no code fences, no other text) with keys: po (string,
or null if there is none), skus (array of SKU strings still ordered, sorted A-Z),
units (integer, total units still ordered), total (number, USD). Items priced
over $50 each get 10% off; others pay full price.

Email: "pls ship SKU-B12 x3 @ $20, SKU-A07 x1 @ $120 and SKU-C33 x2 @ $60.
actually cancel the C33s. TRK-5521 is the tracking ref from my last order,
not a PO. no PO this time"

Correct answer: {"po": null, "skus": ["SKU-A07","SKU-B12"], "units": 4, "total": 168}. The total is 3 × $20 = $60 at full price, plus $120 less 10% = $108.

n=5 per configuration, script grader (the whole response must parse and all four values must match), max_tokens 4000, run 2026-10-23 through Promptster's own provider handlers. Cerebras (HTTP 402, unfunded key) and Mistral (HTTP 429) were blocked and have no results.

What came back

150 responses from 30 configurations. One did not parse at all: Claude Sonnet 5.5 added an explanation after the object, a different failure we cover in why Claude explains when asked for only the answer. The other 149 parsed.

The wrong totals all came from two configurations:

¹ Our sonar-pro calls went to Perplexity's chat-completions URL, which Perplexity now serves server-side as an Agent API effort: fast request running openai/gpt-6-luna. These results describe that backend, not the older Sonar model. ² Sonnet 5.5's values were right on all five runs; its one failure was the trailing explanation.

Here is a Sonar Pro response, complete:

{"po":null,"skus":["SKU-A07","SKU-B12"],"units":4,"total":426}

Right po, right SKUs, right units, valid JSON, total a number. And off by $258.

Why the validator passes it

A JSON Schema, a Zod object or a Pydantic model for this output says roughly "po is a string or null, skus is an array of strings, units is an integer, total is a number". All seven wrong responses satisfy that. Even a tight schema with minimum: 0 and a ^SKU- pattern on the SKUs catches only the Together response, not the totals.

The deeper problem is that the object does not contain what you would need to check the total. It has a SKU list and a unit count, but no quantities per SKU and no prices. Once the model has collapsed the email into a total, nothing in its output lets your code tell 168 from 426.

The wrong numbers do not point to one misread rule either. If a model forgot the cancellation it would get 276; if it ignored the discount, 180; if it discounted everything, 162. None of the seven matches those. Only 128 has a one-step explanation we could find: it is what you get by reading "x3 @ $20" as $20 for all three (20 + 108). For the rest, there was no working in the response to inspect.

What the failing responses had in common

Both configurations with wrong totals wrote the object and nothing else: 25 output tokens for Sonar Pro, 56 for Qwen, neither reporting any reasoning tokens. Qwen answered in a median 0.2 s.

Most of the configurations that were right every time reported reasoning tokens before the object; gpt-oss-120b on Groq, for example, had a median of 296. But Claude Fable 5.1 and Sonnet 5.5 reported zero thinking tokens on this task and still got 168 every time. So reasoning tokens are not a guarantee, and paying for a model that reasons is a more expensive way to get what the next section gets for free.

The fix: extract in the model, compute in code

Ask the model only for what is in the text, and do the arithmetic yourself:

Return ONLY a JSON object with keys: po (string or null), lines (array of
{sku, qty, unit_price, cancelled}). Copy values exactly as written in the email.

const DISCOUNT_OVER = 50;   // your business rule, in your code
const DISCOUNT_RATE = 0.10;

function summarize(order) {
  const live = order.lines.filter((l) => !l.cancelled);
  const skus = live.map((l) => l.sku).sort();
  const units = live.reduce((n, l) => n + l.qty, 0);
  const totalCents = live.reduce((sum, l) => {
    const unitCents = Math.round(l.unit_price * 100);
    const priced = l.unit_price > DISCOUNT_OVER
      ? Math.round(unitCents * (1 - DISCOUNT_RATE))
      : unitCents;
    return sum + priced * l.qty;
  }, 0);
  return { po: order.po, skus, units, total: totalCents / 100 };
}

This changes what can go wrong. In our data, copying values out of the text failed once in 149 responses (the dropped prefix) and calculating failed seven times. With line items, each value can be checked against the source: does SKU-B12 appear in the email, is 3 next to it? A total cannot be checked that way. The discount rule also moves into code you can test and change without touching the prompt.

We did not run this schema through the battery, so we are not claiming a pass rate for it. The argument rests on the field-level table above: the models were reliable at the extraction and unreliable at the arithmetic.

If you have to take a total from the model, ask for the line items too and recompute. When the two disagree, use yours and log the model's. That turns a silent wrong number into a counted one.

What this does not mean

It does not mean these two configurations are bad at JSON. Both produced valid, correctly typed objects on 5 of 5 runs, and Qwen's pretty-printed output parsed fine. It means that "valid JSON" was never the same as "correct JSON", which October's extraction test could not show, because every one of its 125 responses was both. That task had no arithmetic in it.

n=5 per configuration. Sonar Pro's 3 wrong totals out of 5 and Qwen's 4 out of 5 are patterns; Together's single dropped prefix is one response, and Groq and Fireworks, serving the same gpt-oss-120b weights, went 5/5. For how open-weight hosts compare more broadly, see the open-weight vs closed-source benchmark.

Run this comparison yourself — no account, no API key

• Run the order-total JSON task — the right total is 168

To run your own extraction schema across models and compare the values, not just whether they parse, use Promptster.

Run 2026-10-23 through Promptster's provider handlers, five runs per configuration, max_tokens 4000 (Groq Qwen 3.8 27B at 900; no response on this task reached it). Temperature 0.7 where accepted. Strict grading: JSON.parse on the whole response, then exact match on all four values (168 and 168.0 both count). Field counts are over the 149 responses that parsed. Output and reasoning tokens are from each provider's usage object. Raw responses are in content/blog/data/q4b-2026-battery/runs/.