Same Prompt, Different Output: temperature=0 Doesn't Mean What You Think

The standard advice for reproducible LLM output is "set temperature to 0." We measured whether that works.

It depends entirely on the provider, and on the newest models you can't set temperature at all.

Five identical requests each

Same prompt, same parameters, same session, byte-compared.

GPT-4.1-nano at temperature 0 returned the identical string five times. That's the behaviour the advice promises.

Gemini 3.6 Flash at temperature 0 returned five different strings. Not slightly different — five distinct outputs. Temperature 0 constrains sampling, but it doesn't eliminate the other sources of variation in a distributed serving stack: batching, hardware scheduling, floating-point non-associativity across parallel reductions. Any of those can flip a token, and one flipped token changes everything after it.

Claude Opus 5 has no temperature parameter at all. It's removed on Claude 5 and Opus 4.7+ — sending it returns a 400. There is no determinism lever to pull.

This isn't theoretical — it changed our results repeatedly

Over a month of benchmarking, variation of this kind produced three results that looked like findings and weren't.

A constraint test that ranged 0/5 to 4/5. We ran the same prompt against Claude Opus 5 in five separate sessions: 1/5, 3/5, 4/5, 0/5, 5/9 — 13 clean runs out of 29. Any single five-trial run would have supported a confident and wrong conclusion about the model.

Identical weights scoring differently. Kimi K3 runs on Together and Fireworks with the same open weights at the same price. On the same constraint test one scored 5/5 and the other 4/5. There is no mechanism by which a host changes a model's ability to avoid a letter — that gap was pure sampling noise, and it's the same size as most differences reported in five-trial comparisons.

26% cost variance on identical prompts. Four measurements of Claude Opus 5 on the same battery: $0.035025, $0.037125, $0.037225, $0.044200. Different token counts, same input.

What to do about it

Don't assume temperature: 0 gives reproducibility. Test it. Five identical requests and a byte comparison takes two minutes and tells you which regime you're in. The answer differs by provider and can change when a provider updates its serving stack.

Never gate on exact output matching. Assert on properties instead: does it parse, does it match the schema, does it contain the required fields, does it satisfy the constraint. Every benchmark in this series is graded that way, which is why the results survive nondeterminism.

Size your samples to the effect you're claiming. If two models differ by less than about 25% on a single run, you have not measured a difference — that's inside our observed variance. For a pass-rate comparison, five trials cannot separate a 50% model from a 90% one. Ten is a floor, not a luxury.

Measure pass rate, not pass/fail. A model that satisfies your constraint 47% of the time isn't "broken" or "working" — it has a rate, and that rate is your real cost multiplier.

Treat published single-run comparisons with suspicion, including the ones early in this series. If a benchmark doesn't say how many trials it ran, assume one, and assume any gap under 25% is noise.

The uncomfortable implication

Reasoning models removed the parameter people used for reproducibility, and the models that still accept it don't all honour it. The industry moved toward less determinism, not more, and the tooling advice hasn't caught up.

Practically, that means the old model of "pin the temperature, snapshot the output, diff against it in CI" is finished for current frontier models. What replaces it is property-based assertions and measured pass rates — which is more work, and is the only approach that actually holds.

Try it yourself

• Run the same prompt a few times — no account needed; watch whether the wording changes

To run the same prompt repeatedly across models and compare variation, that's Promptster.

Determinism measured 2026-08-11 against the OpenAI, Google and Anthropic APIs directly, five identical requests per configuration in a single session, outputs compared by SHA-256 of the trimmed string. max_tokens: 300. The variance examples are drawn from runs across this series, dated in their own posts. Claude Opus 5 rejects an explicit temperature, so it was called without one — its variation is the model's default behaviour, not a temperature setting.