Kimi K3 vs the Closed Frontier — and the Same Model Runs 2x Faster on One Host Than the Other

Moonshot AI released Kimi K3 in July — 2.8 trillion parameters, open weights, roughly a million tokens of context. It's served by multiple inference providers, and unusually, two of them price it identically: Together and Fireworks both charge $3 per million input tokens and $15 per million output.

That coincidence makes a comparison possible that you almost never get cleanly: the same weights, at the same price, on two different hosts. Any difference is infrastructure, not the model.

So we ran two experiments at once. Does the open-weight model keep up with the closed frontier? And does it matter who you rent it from?

It keeps up

Same hard battery as the rest of this series — three tasks, each graded by script against a known answer.

An open-weight model you can download matched two proprietary flagships on all three, including the letter-counting task that has been reliably embarrassing frontier models for years.

That's the fifth consecutive post in this series where the hard battery failed to separate anything. At this point that isn't a surprise — it's the finding. On ordinary verifiable work, the frontier has converged.

Same model, same price, twice the wait

Fireworks served the identical model exactly 2.00× faster, for effectively the same money — the small cost difference is just token-count variation between runs, since the per-token rates match to the cent.

Nothing about the model explains this. Same weights, same context window, same published price. The entire gap is serving infrastructure: batching, hardware, queue depth, how aggressively the host schedules concurrent work.

If you've chosen an open-weight model and stopped there, you've made half the decision. On this workload the host choice was worth more than most model choices we've measured in this series.

The cheaper per-token model cost more

Here's the thread that keeps recurring across this series. Kimi K3's output rate is $15 per million. GPT-5.6 Sol's is $30 — double.

K3 cost 1.54× more than Sol despite charging half the rate, because it used 3.1× more output tokens for identically-graded answers.

We've now measured this three ways in this series: Luna's 5× list discount became 3.3× because it's verbose, Grok's 4.2× became 11.4× because it's terse, and here a model with half the output rate ends up 54% more expensive. Per-token price tells you very little about what you'll actually pay. Measure tokens on your own prompts.

The result that undermines our own methodology

We ran the no-letter-"e" constraint test five times per configuration:

Look at the first two rows. That is the same model, with the same weights, scoring 5/5 and 4/5.

There is no possible mechanism by which one host's copy of Kimi K3 is better at avoiding a letter than another's. The difference is sampling noise, full stop — and it's the same magnitude as most of the "differences" you'll read about in five-trial model comparisons, including several we've published in this series.

We've been saying since the Fable 5 post that five trials can't rank two models. This is the clean demonstration: a configuration that is provably identical to another produced a different score.

Opus 5's 0/5 is worth its own note. Across four separate runs of this identical test, Opus 5 has scored 1/5, 3/5, 4/5, and now 0/5 — eight clean runs out of twenty. That spread isn't a measurement of the model so much as a measurement of how unstable this behaviour is. If your product depends on a hard output rule, the lesson isn't "pick a different model." It's that you must validate the output, because the best runs and the worst runs come from the same model on the same prompt.

What to take away

Open weights are competitive for one-shot production work. Kimi K3 matched two closed flagships on every task we could construct. If licensing, portability, or self-hosting matter to you, this battery gives you no performance reason to avoid it.

Choose the host deliberately. Identical model, identical price, 2× the latency. That decision is invisible on any model leaderboard and it was the largest single effect in this experiment.

Budget from measured tokens, never from the rate card. Half the per-token price, 54% higher bill.

Treat small-sample comparisons — ours included — as directional. We got two different scores from the same weights. Any ranking built on five trials is reporting noise with a confident face on it.

Run it yourself

No account, no API key:

• The letter-counting task — answer is 10
• The constraint test — run it a few times and watch it vary

To compare hosts and models side by side on your own prompts, with the token counts that drive the bill, that's Promptster.

Run 2026-08-08 via the Promptster API, max_tokens: 4000. Kimi K3 tested as moonshotai/Kimi-K3 on Together and accounts/fireworks/models/kimi-k3 on Fireworks; both list at $3/$15 per million input/output tokens. Opus 5 $5/$25, GPT-5.6 Sol $5/$30. The three graded tasks were run once per configuration — enough for pass/fail against a deterministic answer, not enough to characterise latency variance, so treat the timings as indicative of a single run rather than a steady-state benchmark. Latency includes network round-trip from a single region and both hosts were measured in the same session. The constraint task was run five times per configuration, which — as the Together/Fireworks split above demonstrates — is not enough to rank anything.