PaperfootJoin free
arXiv

Rubric content controls how strictly AI judges grade

Rubric content moved the pass rate of correct outputs from 2% to 100%, whereas changing the judge model moved it by a few points.

Paper by Saeid Asgari, Rui Ying, Sara Malvar, 4 more authorsRanveer Chandra
Preprint: empirical study

Edited by Boris Djordjevic · Paperfoot

Paper

This preprint presents a study of checklist rubrics used by large language models to grade outputs on a controlled task with planted flaws and two human-labeled datasets, SummEdits and RAGTruth. Holding the judge and outputs fixed, rubric content moved correct outputs’ pass rate from 2% to 100%. Changing the judge model or how the same checks were presented moved it by a few points. Models reviewing rubrics added unstated requirements and deleted human-taught checks across three reviser models and all three datasets.

For rubrics requiring every check to pass, the authors showed an exact formula assigning checks credit for catching flaws and blame for wrongly failing correct outputs. Self-review deleted checks making unique catches and kept checks causing false failures. A change-control gate combining these scores with protection for label-derived checks held nearly every drifting revision chain at its starting quality, with identical decisions under three judge models.

Why it matters

In these grading tests, checklist content affected correct outputs’ pass rates far more than changing the judge model did. The attribution method gave researchers a way to identify checks that caught otherwise-missed flaws and checks that wrongly failed correct outputs.

Caveats

The attribution formula applies specifically to rubrics where all checks must pass. The tests covered a controlled grading task and two human-labeled datasets, and the work is a preprint that has not yet been peer-reviewed.

From the paper

Limitation

The Recon environments and labels are synthetic and ours; the SummEdits and RAGTruth replications cover consistency and faithfulness grading with human labels but not agentic tasks.

Limitations.

Quoted word for word from the full text on arXiv.

In this paper

checklist rubric evaluation for LLM judges

Methods
Shapley attribution (this paper)Gate S+P (this paper)Gate S (this paper)Gate P (this paper)Gate R (baseline)Claude Haiku 5.5Sonnet 5.5Opus 5.5
Datasets
SummEdits (evaluated)RAGTruth (evaluated)Recon-72 (this paper)Recon-108 (this paper)

What it shows

Holding the judge and outputs fixed, rubric content moves the pass rate of correct outputs from 2% to 100%, whereas changing judge model or rendering changes it by only a few points.

Holding the judge and the outputs fixed, rubric content moves the pass rate of correct outputs from 2% to 100%, whereas changing the judge model or the rendering of the same checks moves it by a few points.

Abstract

Under all-must-pass aggregation, the Shapley value of each check has an exact closed form, giving attribution quantities in linear time.

To measure and control this, we show that under all-must-pass aggregation the Shapley value of each check has an exact closed form, which yields per-check catch credit, false-fail blame and unique catches in linear time.

Abstract

Model self-review drifts checklist rubrics toward the reviser's standard, deleting label-taught checks and retaining checks that cause false fails.

Attribution shows that self-review deletes the checks that do unique work and keeps the checks that cause false fails.

Abstract

Combining Shapley attribution with a provenance freeze preserves rubric quality across drifting revision chains under multiple judge models.

A change-control gate that combines these quantities with a provenance freeze on label-derived checks holds nearly every drifting revision chain at its starting quality, with identical decisions under three judge models.

Abstract

Quoted word for word from the full text on arXiv.

The paper

Rubrics Carry the Standard: Shapley Attribution for Checklist-Based LLM Judges

Saeid Asgari, Rui Ying, Sara Malvar, 4 more authorsRanveer Chandra

arXiv, Oct 18, 2026, Open access, CC BY, Preprint, not peer-reviewed

doi.org/10.48550/arxiv.2610.24971

arXiv:2610.24971v1 in cs.CL

AbstractPDFHTML

Cite as BibTeX
@misc{asgari2026rubrics,
  title = {Rubrics Carry the Standard: Shapley Attribution for Checklist-Based LLM Judges},
  author = {Asgari, Saeid and Ying, Rui and Malvar, Sara and Agarwal, Rakshanda and Sun, Bruce and Wang, Claire and Nunes, Leonardo de Oliveira and Chandra, Ranveer},
  year = {2026},
  eprint = {2610.24971},
  archivePrefix = {arXiv},
  primaryClass = {cs.CL},
  doi = {10.48550/arxiv.2610.24971},
  url = {https://arxiv.org/abs/2610.24971}
}

Related