Open-Weight Models for Private Deployment, November 2026: Which Ones Held Up on a Harder Battery

By Promptster Team · 2026-11-12

If your data can't leave your network, you are choosing between open-weight models. Which of them are good enough to deploy privately? This post updates our April comparison of models for data privacy and local hosting with November data. That post covers the privacy options themselves, and we don't repeat them here.

On four harder graded tasks, six of the seven open-weight models we could run passed 20/20 on at least one host: gpt-oss-120b, Kimi K3, GLM 5.3, DeepSeek V4.1 Flash, DeepSeek V4 Pro and MiniMax M3. The exception was Qwen 3.8 27B, which passed 6/20, and five of its 14 misses were caused by the on-demand tier cap of 1,000 output tokens per minute on Groq rather than wrong answers. One caveat applies to every row: every number here comes from a hosted API, not a self-hosted run. Your own deployment may use different quantization, serving software and defaults, so treat this as a shortlist to retest, not a prediction.

How we tested

Four script-graded tasks, each with a trap, and no LLM judge:

We ran each task five times per configuration (n=5) with temperature 0.7 and max_tokens 4000, except Qwen 3.8 27B on Groq at 900 (explained below). The runs were on 2026-10-23, through Promptster's own provider handlers. Costs are cost_usd at list rates from shared/pricing.ts. Latency is wall-clock time from one machine.

The tasks are short, single-turn and verifiable, which is similar to the extraction work many private deployments handle. They don't test long documents, multi-step agents or tool use.

The results

Model Host (hosted API) Passed Cost, 20 calls Median latency
gpt-oss-120b Groq 20/20 $0.0069 1.0 s
gpt-oss-120b Fireworks 20/20 $0.0076 1.6 s
gpt-oss-120b Together 19/20 $0.0072 4.0 s
DeepSeek V4.1 Flash Fireworks 20/20 $0.0119 3.2 s
MiniMax M3 Fireworks 20/20 $0.0148 2.4 s
DeepSeek V4 Pro (-0813) Together 20/20 $0.0391 2.1 s
GLM 5.3 Fireworks 20/20 $0.0568 6.7 s
Kimi K3 Together 20/20 $0.1551 6.2 s
Kimi K3 Fireworks 20/20 $0.1601 10.3 s
Qwen 3.8 27B Groq (max_tokens 900) 6/20 $0.0220 0.2 s
gpt-oss-120b, Qwen 3.8 27B Cerebras blocked: HTTP 402 — —

The Cerebras key was unfunded, so its first call returned HTTP 402 payment_required and the remaining calls were skipped. We have no Cerebras numbers and have not estimated any.

On the first pass, Together's Kimi K3 timed out at 120 s on 2 of its 20 calls. Both passed when rerun, and the table shows the reruns.

Qwen 3.8 27B: 6/20, and why that is not one number

Qwen 3.8 27B was the only open-weight model that lost more than one call. Its misses fall into three groups.

Five were the cap. Groq applies the on-demand tier cap of 1,000 output tokens per minute to this model, and a larger request is rejected with HTTP 429, so we ran it at max_tokens 900. All five stock-floor runs stopped at exactly 900 tokens with finish_reason: length, partway through the working. That is a 0/5 for the limit, not for the model's arithmetic. A self-hosted deployment would not have that cap.

Six were wrong values. On json-order, the JSON was valid, the PO was correctly null and the SKUs were right, but the total was wrong four times: 128, 198, 198 and 150 (the correct answer is 168). On launch-notes, two runs got the right date and the wrong budget: Nov 21, $42500 and Nov 21, $35000.

Three were the word count. Three of the five tagline runs broke only the seven-word rule. The diagnostic shows they followed every other rule. One run produced:

refill your flask to stay hydrated
one refill keeps your drink cold
refill often to save on costs

These responses were also very short. Groq reported no reasoning tokens for Qwen, and it answered json-order in 56 output tokens, while the other open-weight models used a few hundred, mostly reasoning. That brevity is why it was the fastest model in the table, at 0.2 s. It is also the most likely reason it miscalculated a discount that every reasoning model got right. That last point is our interpretation; the data doesn't show the cause directly. Groq also returned 429 on 3 tagline calls on the first pass; all three succeeded on rerun.

We could not test Qwen 3.8 27B on a second host, because the only other one we have, Cerebras, was blocked. So we can't separate the model from Groq's configuration of it, and we would not rule it out on this data alone.

The same weights, three hosts, one slip

gpt-oss-120b passed 20/20 on Groq and on Fireworks and 19/20 on Together. In the one miss, on json-order, Together's response dropped the prefix from the SKUs:

{"po":null,"skus":["A07","B12"],"units":4,"total":168}

That is one response in five, so we don't read a trend into it. It still matters for self-hosting: the same weights did not produce identical results across hosts. A self-hosted deployment is another host, with its own serving stack. Before you rely on these results, rerun the same tasks on your own deployment. Our gpt-oss host comparison has the full breakdown.

What changed since April

The April post recommended Llama 4 Scout and Mistral Large for maximum control. Neither is in this battery: Groq has since retired Llama 4 Scout, and our only Mistral configuration was blocked by HTTP 429 during this run. In October our easier battery didn't separate any of the open-weight models it included (October's comparison). The harder tasks still separated only one open-weight model, Qwen 3.8 27B, and only part of that gap was the model.

Cost also tells a different story from April. The cheapest 20/20 open-weight result was gpt-oss-120b on Groq at $0.0069 for all 20 calls, and the most expensive was Kimi K3 on Fireworks at $0.1601. That is a 23× spread for the same score. Those are hosted per-token prices, not GPU costs.

What we'd do

Shortlist from the 20/20 group, starting with the cheapest that fits your hardware. On these tasks, gpt-oss-120b, DeepSeek V4.1 Flash and MiniMax M3 matched Kimi K3 and GLM 5.3 at a fraction of the hosted cost. Model size and hardware requirements are not in this data. Check each model's own page before you plan capacity.

Test on a hosted API first, then on your own stack, with the same graded prompts. Our data covers the first step only. The Together gpt-oss slip shows why the second step matters.

Don't drop a model because of a host limit. Qwen 3.8 27B's 0/5 on stock-floor came from the on-demand tier cap of 1,000 output tokens per minute on Groq. Run it under your own limits before you decide.

Run this comparison yourself — no account, no API key

To compare open-weight models across hosts on your own prompts before you commit to hardware, use Promptster. For the broader open vs closed picture, see the original open-source vs closed-source benchmark.


Run 2026-10-23 through Promptster's provider handlers, on hosted APIs only. No model was self-hosted for this post. Configurations: gpt-oss-120b on Groq, Together and Fireworks; Kimi K3 on Together and Fireworks; GLM 5.3, DeepSeek V4.1 Flash and MiniMax M3 on Fireworks; DeepSeek V4 Pro (deepseek-ai/DeepSeek-V4-Pro-0813) on Together; Qwen 3.8 27B on Groq at max_tokens 900; Cerebras gpt-oss-120b and Qwen 3.8 27B returned HTTP 402. Four script-graded tasks, five runs each, temperature 0.7, max_tokens 4000 unless noted. Costs are cost_usd from list rates in shared/pricing.ts. Latency is wall-clock from one machine. Raw responses and usage objects for every call are in content/blog/data/q4b-2026-battery/.