A method tracks how many rigged votes an AI leaderboard withstands
In replays on 1.8 million Chatbot Arena votes, a few hundred rigged votes made standard confidence intervals certify false orderings, while the new method stayed valid.
Edited by Boris Djordjevic · Paperfoot
Researchers developed the certified corruption budget, a method for certifying pairwise claims about AI models, and tested it on 1.8 million votes from Chatbot Arena, a model leaderboard. For each pairwise claim, with probability at least one minus a specified error rate, the claim is correct or corruption exceeds its budget, simultaneously at all times.
The authors showed that forged records and votes altered after being seen require different certificates. A forgery certificate failed with probability approaching one against an attacker flipping votes it had seen; charging roughly twice as much per record remained valid. In replays, a few hundred rigged votes made standard confidence intervals—statistical ranges expressing uncertainty—certify false orderings, while the new certificate stayed valid. On real votes, the certificate showed that clearly separated models could withstand about 2,000 forged votes.
Why it matters
The method lets leaderboard hosts publish a corruption tolerance with each pairwise claim, rather than assuming the records are genuine. Its mathematical guarantee covers attackers who watch every certificate, without assuming a limit on how many records they corrupt.
Caveats
The abstract does not report tests on other types of leaderboards or benchmarks. The paper is a preprint that has not yet been peer-reviewed.
In this paper
anytime-valid leaderboard claims under adaptive rigging
- Methods
- certified corruption budget (this paper)
- Datasets
- Chatbot Arena (evaluated)
Results
- certified corruption budget, forged votes withstood, on Chatbot Arena, clearly separated models: about 2,000 forged votes
On real votes, our certificate shows that clearly separated models withstand about 2,000 forged votes.
Abstract
What it shows
- Certified corruption budgets ensure that with probability at least 1-α, simultaneously at all times, the claim is correct or more than B_t records were corrupted.
With probability at least 1-α, simultaneously at all times, the claim is correct or more than B_t records were corrupted.
Abstract
- The certificate for forgeries fails against an attacker who flips seen votes, while one charging roughly twice as much remains valid.
… the certificate for forgeries fails, with probability approaching one, against an attacker who flips votes it has seen, while one that charges roughly twice as much per record remains valid, with constant bets even against attackers who see the future, and no smaller charge is valid at every level …
Abstract
- With win fraction 1/2+δ, each new record adds close to 2δ to the number of forged records the claim withstands (δ flipped).
The certified budget grows nearly as fast as any valid method allows: with a win fraction 1/2+δ, each new record adds close to 2δ to the number of forged records the claim can withstand (δ flipped).
Abstract
- Publishing the best of V private variants costs only an amount growing like log V.
Publishing the best of V private variants costs only an amount growing like log V.
Abstract
Quoted word for word from the abstract on arXiv.
The paper
Certified Corruption Budgets: Anytime-Valid Leaderboard Claims under Adaptive Rigging
arXiv, Oct 6, 2026, Open access, Preprint, not peer-reviewed
doi.org/10.48550/arxiv.2610.10597
arXiv:2610.10597v1 in cs.CR, also cs.AI, stat.ME
Cite as BibTeX
@misc{khosravi2026certified,
title = {Certified Corruption Budgets: Anytime-Valid Leaderboard Claims under Adaptive Rigging},
author = {Khosravi, Hamed and Huo, Xiaoming},
year = {2026},
eprint = {2610.10597},
archivePrefix = {arXiv},
primaryClass = {cs.CR},
doi = {10.48550/arxiv.2610.10597},
url = {https://arxiv.org/abs/2610.10597}
}