Language models used as judges change verdicts and repeat errors
An audit tested six frontier models across four benchmarks and found the most consistent judge agreed with ground truth only 51% of the time.
Edited by Boris Djordjevic · Paperfoot
The preprint presents a reliability audit of language models used as judges, stress-testing six frontier models across four benchmarks, five prompt formats, two presentation orders, three sampling temperatures, and ten repetitions per condition. The tests revealed severe vulnerabilities. Verdicts changed across identical runs even at temperature zero, a setting meant to eliminate randomness. Swapping presentation orders flipped the majority of verdicts on challenging tasks. The most deterministic judge achieved consistency by trivially repeating incorrect verdicts, agreeing with ground truth only 51% of the time.
To formalize these failures, the authors introduced the trustworthy verdict rate, a metric measuring whether an evaluation is reproducible, order-invariant, and accurate. They derived an upper bound on accuracy from position bias and showed that reliability depends on the item rather than the model. Shifting from pairwise win rates to holistic rubric scoring improved trustworthiness more than any single-format prompting intervention.
Why it matters
Because researchers widely rely on language models to evaluate artificial intelligence systems, the findings show that treating these judges as deterministic ground truth leads to brittle and misleading results. The audit across six frontier models and four benchmarks shows where automated evaluation fails and provides a rubric-based method to make testing more trustworthy.
Caveats
The abstract does not name the six frontier models or the four benchmarks evaluated in the audit. The work is also a preprint that has not yet been peer-reviewed.
From the paper
- Limitation
First, due to computational tractability, we restricted our evaluation to four English-only benchmarks.
Quoted word for word from the full text on arXiv.
In this paper
LLM judge reliability evaluation
- Methods
- trustworthy verdict rate (T) (this paper)GPT-5.2 (evaluated)GPT-4o (evaluated)Gemini-2.5-Pro (evaluated)Gemini-2.0-Flash (evaluated)Claude-Sonnet-4.6 (evaluated)Llama-3.3-70B (evaluated)FLASK (evaluated)G-Eval (evaluated)
- Datasets
- MT-Bench (evaluated)AlpacaFarm (evaluated)JudgeBench (evaluated)LLMBar (evaluated)
Results
- Gemini-2.0-Flash, flip rate, on JudgeBench, Verdict-first XML: 98.0%; Gemini-2.0-Flash, observed accuracy, on JudgeBench, Verdict-first XML: 51.0%
Its flip rate is \varphi=98.0%, implying a maximum attainable accuracy of 51.0%, and the observed accuracy is exactly A=51.0%.
5.1 The Position-Bias Ceiling is Tight
- Gemini-2.0-Flash, T, on JudgeBench, Verdict-first XML: 2.0%; Gemini-2.0-Flash, C, on JudgeBench, Verdict-first XML: 100%
Under Equation (4), this yields T=2.0%: the judge is perfectly reproducible across repeated calls ( C=100% ), but its verdicts carry essentially no trustworthy signal beyond order-induced coin-flipping.
5.1 The Position-Bias Ceiling is Tight
- Gemini-2.0-Flash, flip rate, on JudgeBench, Verdict-first JSON: 82.0%; Gemini-2.0-Flash, observed accuracy, on JudgeBench, Verdict-first JSON: 52.0%; Gemini-2.0-Flash, T, on JudgeBench, Verdict-first JSON: 10.8%
The same pattern appears, though less extremely, for Gemini-2.0-Flash with Verdict-first JSON on JudgeBench: \varphi=82.0% implies a ceiling of 59.0%, while the observed accuracy is 52.0% and T falls to 10.8%.
5.1 The Position-Bias Ceiling is Tight
- Gemini-2.5-Pro, A, on JudgeBench, Verdict-first JSON: 88.5%; Gemini-2.5-Pro, T, on JudgeBench, Verdict-first JSON: 77.5%
On JudgeBench, Gemini-2.5-Pro improves from A=63.5%,T=28.9% under Direct prompting to A=88.5%,T=77.5% under Verdict-first JSON.
5.2 T Reveals Reliability Differences Hidden by Accuracy
- Gemini-2.5-Pro, position flips, on JudgeBench, Verdict-first JSON: 6.9%
The 25% accuracy gain is accompanied by a 49% gain in T, because the format change both improves accuracy and reduces position flips from \varphi=56.0% to \varphi=6.9%.
5.2 T Reveals Reliability Differences Hidden by Accuracy
- Gemini-2.5-Pro, T, on JudgeBench, G-Eval: 77.9%
G-Eval elicits all criterion scores in a single prompt, preserving C while reducing position sensitivity; for capable judges this can lift T substantially (Gemini-2.5-Pro: T 28.9%\to 77.9% on JudgeBench, a +49 pp gain exceeding any format-induced swing in Table 4).
5.6 Rubric-Based Evaluation: FLASK and G-Eval
- GPT-5.2, C, on JudgeBench: 34.8%
FLASK reduces \varphi for some judges but collapses C universally (e.g., GPT-5.2: C 86.0%\to 34.8% on JudgeBench), leaving T lower than Direct for five of six judges.
5.6 Rubric-Based Evaluation: FLASK and G-Eval
What it shows
- Under a forced-binary preference setup, position flip rate imposes a tight upper bound on observable accuracy of 1 - phi / 2 that no capability improvement can exceed.
For our forced-binary setup ( q=\tfrac{1}{2} ), Eq. (3) simplifies to A≤ 1-\varphi/2.
3.4 A Tight Upper Bound on Observable Accuracy
- Stochastic consistency alone does not guarantee reliability, as a judge can achieve 100% consistency while producing trustworthy verdicts only 2% of the time.
Under Equation (4), this yields T=2.0%: the judge is perfectly reproducible across repeated calls ( C=100% ), but its verdicts carry essentially no trustworthy signal beyond order-induced coin-flipping.
5.1 The Position-Bias Ceiling is Tight
- The largest source of variation in trustworthy verdict rate across evaluated cells is the judge by dataset interaction.
A three-factor ANOVA of T across the 120 judge–format–dataset cells shows that the largest source of variation is the judge × dataset interaction ( \eta^{2}=28.1% ), exceeding the judge ( 27.0% ) and dataset ( 26.2% ) main effects (see Appendix C.1).
Judge–dataset interaction.
- Increasing sampling temperature primarily reduces stochastic consistency rather than the bias-adjusted capability term.
As shown in Figure 3, mean C drops by 12.1 points across the sweep, while (A-\varphi/2) changes by only +0.3 points across 48 cells.
Temperature ablation.
- The empirical evaluation is restricted to four English-only benchmarks, closed-weight models without access to log-probabilities, and pairwise preference data.
First, due to computational tractability, we restricted our evaluation to four English-only benchmarks.
Limitations
Quoted word for word from the full text on arXiv.
The paper
All Verdicts are Not Equal: Rethinking LLM Judge Reliability
arXiv, Oct 8, 2026, Open access, CC BY-NC-ND, Preprint, not peer-reviewed
doi.org/10.48550/arxiv.2610.12083
arXiv:2610.12083v1 in cs.CL, also cs.AI
Comments: Accepted at AACL IJCNLP (Main) 2026
Cite as BibTeX
@misc{kumar2026verdicts,
title = {All Verdicts are Not Equal: Rethinking LLM Judge Reliability},
author = {Kumar, Vineet and Rathore, Darshita and Moitra, Anindya},
year = {2026},
eprint = {2610.12083},
archivePrefix = {arXiv},
primaryClass = {cs.CL},
doi = {10.48550/arxiv.2610.12083},
url = {https://arxiv.org/abs/2610.12083}
}
