The 3-Judge Consensus Pattern: How to Cut LLM-as-Judge Bias (With the Math)

Our LLM-as-a-judge bias audit showed a clean 3-for-3 self-preference result: each of three major providers ranked its own response #1 when asked to rank three anonymous outputs. That's an eval methodology's worst nightmare — your "objective" quality score is actually a mirror held up to whichever model you chose as judge.

The fix is simple and deserves its own post: don't use one judge; use three, and aggregate their rankings. Each judge's self-preference cancels against the others', and the aggregated ranking converges toward genuine quality instead of judge-provider identity.

Here's how the math works, on the same data that produced the bias.

The Data

Three responses to "Write a 120-word paragraph on why TDD matters to junior developers":

• Response A — OpenAI GPT-4o
• Response B — Anthropic Claude Sonnet 4.5
• Response C — Google Gemini 2.5 Flash Lite

Three judges ranked them 1 (best) to 3 (worst) with the responses anonymized and in fixed order:

The bias is visible on the diagonal. Each judge ranked its own provider #1.

The Aggregation

Take the average rank each response received across the three judges. Lower average = better:

Aggregated ranking: C > B > A.

No single judge would have produced this answer. Each judge's individual ranking put its own provider at the top. The aggregate puts Google's response at the top — the response that got the second-best rank (2) from two different judges plus a first-place vote from its own.

This is consensus working the way it should: each judge contributes a biased signal, but the biases cancel. What's left is the signal the judges agree on, which correlates better with quality than any individual judge's ranking.

Why This Works (The Intuition)

Each LLM judge has two components to its rating:

• Signal: genuine preference for higher-quality outputs
• Bias: preference for its own provider's style

When you average across three differently-biased judges, the signal adds coherently (all three judges correlate with real quality in roughly the same direction) while the biases are in different directions and partially cancel. This is the same reason averaging N noisy sensors gives you a better reading than one precise sensor.

The key word is differently-biased. Three Claude judges (Opus, Sonnet, Haiku) would not work — all three have the same self-preference bias. You need three judges from different provider families.

The 3-Judge Consensus Recipe

# Pseudocode
responses = [gen(openai), gen(anthropic), gen(google)]

judges = ["openai", "anthropic", "google"]
ranks_by_judge = {}
for judge in judges:
    ranks = judge_rank(judge, responses)  # returns ordering [A, B, C] → ranks [1,2,3]
    ranks_by_judge[judge] = ranks

# Average rank per response
avg_ranks = {
    i: sum(ranks_by_judge[j][i] for j in judges) / len(judges)
    for i in range(len(responses))
}
winner = min(avg_ranks, key=avg_ranks.get)

Three implementation notes that matter:

1. Use different judge families. OpenAI, Anthropic, Google is the natural triad. Adding a fourth judge helps — especially a web-connected one like Perplexity — but three captures most of the benefit.

2. Randomize response order per judge. Our original test didn't — we used a fixed order (A, B, C always = OpenAI, Anthropic, Google). This means some of the diagonal pattern could be position bias. For a proper eval, shuffle the order per judge.

3. Use Kendall's tau or Spearman's rho for confidence. Compute pairwise rank correlation across judges. High correlation = judges agree on the underlying quality signal. Low correlation = the task is subjective enough that no single answer exists.

When 3 Isn't Enough

Three judges gets you most of the benefit but not all of it. For high-stakes evaluations:

• Scale to 5-7 judges across provider families (add DeepSeek, Mistral, xAI as tiebreakers).
• Use blind human evaluators as a calibration set for a sample of outputs — if 5 judges all rank C first but 4 of 5 humans rank A first, you have a systematic gap to study.
• Multi-trial each judge with different response orderings to control for position bias.

For our analysis of correlated hallucinations across models from the same training lineage, see the 11-provider consensus study.

Cost Considerations

Three judges triples the cost of the eval call. For an eval that runs once per PR, this is trivial ($0.003 × 3 = $0.009 per evaluation). For an eval that runs on every production request, it's meaningful.

Pragmatic middle ground: use a single cheap judge for real-time production monitoring, and a three-judge panel for weekly regression runs and release gating. Real-time catches drift; the panel gates quality decisions.

The Wider Pattern

Self-preference is just one bias. LLM judges also exhibit:

• Position bias (prefer responses in position A over C)
• Verbosity bias (prefer longer responses regardless of quality)
• Authority bias (prefer responses that sound confident, even when wrong)
• Style bias (prefer outputs matching their own style conventions)

Multi-judge consensus cancels some of these partially. Full debiasing requires a combination of consensus, position randomization, output normalization, and — ultimately — human calibration. No fully automated eval stack is fully unbiased. The 3-judge pattern is the cheapest, simplest, biggest-impact upgrade available in 2026.

For more on the evaluation landscape, see our LLM-as-a-judge bias audit and the consensus analysis deep-dive.

Analysis based on data from 2026-04-19 judge-bias audit. Aggregation methodology: simple mean of ranks. Alternative aggregations (Borda count, Condorcet) can give different results in tight races; use whichever your downstream decision process supports.