A method tracks how many rigged votes an AI leaderboard withstands
In replays on 1.8 million Chatbot Arena votes, a few hundred rigged votes made standard confidence intervals certify false orderings, while the new method stayed valid.
Edited by Boris Djordjevic · Paperfoot
Researchers developed the certified corruption budget, a method for certifying pairwise claims about AI models, and tested it on 1.8 million votes from Chatbot Arena, a model leaderboard. For each pairwise claim, with probability at least one minus a specified error rate, the claim is correct or corruption exceeds its budget, simultaneously at all times.
The authors showed that forged records and votes altered after being seen require different certificates. A forgery certificate failed with probability approaching one against an attacker flipping votes it had seen; charging roughly twice as much per record remained valid. In replays, a few hundred rigged votes made standard confidence intervals—statistical ranges expressing uncertainty—certify false orderings, while the new certificate stayed valid. On real votes, the certificate showed that clearly separated models could withstand about 2,000 forged votes.
Why it matters
The method lets leaderboard hosts publish a corruption tolerance with each pairwise claim, rather than assuming the records are genuine. Its mathematical guarantee covers attackers who watch every certificate, without assuming a limit on how many records they corrupt.
Caveats
The abstract does not report tests on other types of leaderboards or benchmarks. The paper is a preprint that has not yet been peer-reviewed.
In this paper
anytime-valid leaderboard claims under adaptive rigging
- Methods
- grid tolerance (this paper)insertion certificate (this paper)replacement certificate (this paper)undeflated betting (baseline)per-step robust test (baseline)Wald interval (baseline)
- Datasets
- Chatbot Arena (evaluated)Massive Multitask Language Understanding (MMLU) (evaluated)MMLU-Pro (evaluated)GPQA (evaluated)IFEval (evaluated)Omni-MATH (evaluated)SWE-bench Verified (evaluated)
- Links
- Code
Results
- insertion certificate, probability of error, B=20: at or below 0.048
Under a burst of forged wins (panel a), undeflated betting and the per-step robust test err with probability 0.33 and 0.27 at B=20 and always for B≥ 40, while our insertion certificate stays at or below 0.048.
6.1 Validity against three attacks
- replacement certificate, error probability, losses flipped, B=20: at or below 0.050
With losses flipped (panel b), the insertion certificate fails from B=5 (at B=20, 0.37 with a single bet and 0.94 with the grid) and the replacement certificate stays at or below 0.050.
6.1 Validity against three attacks
- insertion certificate, median \widehat{B}_{t}/t, \delta=0.1 and t=8,000: 0.17; replacement certificate, median \widehat{B}_{t}/t, \delta=0.1 and t=8,000: 0.083
At \delta=0.1 and t=8,000 the median \widehat{B}_{t}/t is 0.17 (insertion) and 0.083 (replacement), against 0.2 and 0.1.
6.2 Growth of the tolerance
What it shows
- The tolerance computed with the replacement cost remains valid against value-dependent and offline attackers, whereas charging only the insertion cost fails against an attacker that flips observed losses to wins.
(b) If c<2c_{\mathrm{ins}}, the attacker in \mathsf{V} that replaces each of the first B uncorrupted losses by a win causes a false statement with probability tending to one as B\to\infty.
Theorem 5.2 (Sharpness).
- No cost below the replacement cost is valid at every error level against value-dependent attackers that flip uncorrupted losses.
This exceeds \alpha for all small enough \alpha, so no cost below c_{\mathrm{rep}} is valid at every level.
Theorem 5.2 (Sharpness).
- For i.i.d. uncorrupted values without ties, the tolerance growth rate per record approaches the optimal upper bound as the maximum bet size tends to zero.
By the last claim of Theorem 5.3 and by Theorem 5.4, \rho lies between q-\bar{\lambda}_{m}/2 (insertion, q=2\delta ) or q-\bar{\lambda}_{m}/6 (replacement, q=\delta ) and q when \bar{\lambda}_{m}≤ 1, so \rho\to q as \bar{\lambda}_{m}\to 0.
Theorem 5.4 (Lower bound).
- Selecting and publishing the best of V private variants costs only an amount scaling with log V.
Publishing the best of V private variants costs only an amount growing like log V.
Abstract
- When all pairs are claimed simultaneously, no certificate separates GPT-4o and Claude-3.5-Sonnet on Chatbot Arena votes.
When all pairs are claimed at once, no certificate separates GPT-4o (2024-05-13) and Claude-3.5-Sonnet (Figure 3 in Appendix C).
6.3 Chatbot Arena and static benchmarks
Quoted word for word from the full text on arXiv.
The paper
Certified Corruption Budgets: Anytime-Valid Leaderboard Claims under Adaptive Rigging
arXiv, Oct 6, 2026, Open access, Preprint, not peer-reviewed
doi.org/10.48550/arxiv.2610.10597
arXiv:2610.10597v1 in cs.CR, also cs.AI, stat.ME
Cite as BibTeX
@misc{khosravi2026certified,
title = {Certified Corruption Budgets: Anytime-Valid Leaderboard Claims under Adaptive Rigging},
author = {Khosravi, Hamed and Huo, Xiaoming},
year = {2026},
eprint = {2610.10597},
archivePrefix = {arXiv},
primaryClass = {cs.CR},
doi = {10.48550/arxiv.2610.10597},
url = {https://arxiv.org/abs/2610.10597}
}