PaperfootJoin free
arXiv

AI judges changed verdicts on identical inputs in cloud tests

Across four judges, verdict flip rates averaged roughly 5% on identical inputs and were about 40% on close calls, despite settings intended for repeatability.

Paper by Krishna Chytanya Ayyagari
Preprint: empirical study

Edited by Boris Djordjevic · Paperfoot

Paper

The empirical study tested four language models acting as judges on Arena-Hard, AlpacaEval 2 and MT-Bench, three standard benchmarks for comparing language-model answers. All ran on one enterprise cloud platform. Researchers repeated identical inputs with a constant platform-reported model version and temperature zero, a setting intended to remove randomness from answer selection.

Per-item verdict flip rates averaged roughly 5% and were about 40% on close calls that decide leaderboard margins. Rates ranged from 0.13% to nearly 10% across judges. The main judge’s aggregate ranking stayed stable. Yet roughly one-fifth to three-quarters of neighboring leaderboard positions could not be statistically distinguished. The authors attributed this limited precision mainly to the finite sample of prompts, rather than the judge. They introduced instability measures and recommended repeated judging, uncertainty intervals and comparisons across model families.

Why it matters

These benchmark tests showed that a stable overall ranking can still overstate how clearly neighboring models are separated. Reporting that uncertainty would help readers interpret leaderboard differences.

Caveats

The tests covered four judges on one cloud platform, not cloud services generally. The work is a preprint and has not been peer-reviewed.

The paper

Pinned and Still Unstable: Within-Judge Verdict Variance and the Noise Floor of LLM-as-Judge Leaderboards

Krishna Chytanya Ayyagari

arXiv, Sep 27, 2026, Open access, CC BY, Preprint, not peer-reviewed

doi.org/10.48550/arxiv.2609.33044

arXiv:2609.33044v2 in cs.CL

AbstractPDFHTML

Comments: v2: corrects reporting details (claim provenance, candidate generation cap, top-K ties, data completeness, abort threshold, model aliases) and clarifies wording; no new experiments, within-judge variance results unchanged. Changes are listed in Appendix D. 32 pages, 3 figures

Cite as BibTeX
@misc{ayyagari2026pinned,
  title = {Pinned and Still Unstable: Within-Judge Verdict Variance and the Noise Floor of LLM-as-Judge Leaderboards},
  author = {Ayyagari, Krishna Chytanya},
  year = {2026},
  eprint = {2609.33044},
  archivePrefix = {arXiv},
  primaryClass = {cs.CL},
  doi = {10.48550/arxiv.2609.33044},
  url = {https://arxiv.org/abs/2609.33044}
}

Related