Written rules miss key human preferences across many tasks
Across 317 million human judgments in seven domains, written rubrics and automated tests left gaps in capturing human preferences compared with densely trained models.
Edited by Boris Djordjevic · Paperfoot
The preprint introduces CreativePreferences, a dataset of 2.8 million texts evaluated by 317 million human preference judgments across 42 benchmark tasks in seven creative domains. The authors modeled these labels using executable programs that verify specific rules, banks of written rubrics, and densely trained models that learn preferences directly.
Across all seven domains, the authors found gaps where written rubrics and automated checks failed to capture human preferences. These gaps appeared even in fields often treated as verifiable, including mathematics, software engineering, news, and peer review. Peer review and creative writing showed the largest gaps between written rubrics and human choices. The gaps also widened as more people took part in a judgment. Furthermore, on texts written by humans, the densely trained models matched human choices more closely than the articulated criteria did.
Why it matters
The results show that in an analogy to Goodhart's law, articulating a preference shifts it away from tacit dimensions on the evaluated tasks. The authors use these gaps to provide recommendations on when tasks can be guided by prompts and when judgments should remain with humans.
Caveats
The paper is a preprint that has not yet been peer-reviewed. In addition, the upper and lower bounds for the gaps rely on the authors' statistical estimation method to infer the value of undiscovered metrics.
From the paper
- Limitation
The full model is one full-data run per task, not a seed ensemble, and the clean-eval AUCs are single-seed point estimates.
Quoted word for word from the full text on arXiv.
In this paper
modeling human preference judgments and estimating articulability and verifiability gaps
- Methods
- articulated model (evaluated)full model (evaluated)AutoMetricsExperiGenHypoGeniCHypotheSAEsLlama-3.1
- Datasets
- CreativePreferences (this paper)Dolma (evaluated)
Results
- articulated model, score, stratified by agency: .612; articulated model, score, frozen cross-domain GroupKFold design: .524
For the agency-agreement label on notice-and-comment data, the articulated model scores.612 when the grouped folds are stratified by agency and.524 under the frozen cross-domain GroupKFold design.
Estimator sensitivity.
- full model, score, evaluation split: .566; full model, score, test split: .639
Its full model has the same problem:.566 on the evaluation split and.639 on the test split, where every other task has evaluation and test within noise of each other.
Estimator sensitivity.
What it shows
- Peer review and creative writing exhibit the largest articulability gaps among the studied domains, which widen as more people participate in judgments.
The size of the gap varies based on domain (e.g. peer review and creative writing have the largest articulability gaps) and widens as more people take part in the judgment, consistent with Collins' collective tacit knowledge.
Abstract
- Full models match human preferences more closely than articulated criteria on human generations, with preference articulation shifting preferences away from tacit dimensions.
We show two consequences: (1) on human generations, the full model more closely matches human preferences, often in disagreement with articulated criteria, and (2) in an analogy to Goodhart's law, articulating preference shifts it away from the tacit dimension.
Abstract
- The observed performance gap cannot be proven to be tacit knowledge as held by a person.
It is not shown to be tacit knowledge as a person holds it.
Whether this is tacit knowledge.
- The observed gap is not separated into somatic and collective kinds of tacit knowledge.
It is not separated into Collins’s somatic and collective kinds; although the fact that more people passing judgement correlates with a bigger gap points to CTK, nothing explicitly tests this.
Whether this is tacit knowledge.
Quoted word for word from the full text on arXiv.
The paper
Verifiable, Articulable, and Tacit Components of Preference
arXiv, Oct 2, 2026, Open access, CC BY, Preprint, not peer-reviewed
doi.org/10.48550/arxiv.2610.03025
arXiv:2610.03025v2 in cs.AI, also cs.CL, cs.CY, cs.LG
Comments: 15 pages main text, 14 pages of references, 107-page appendix (136 pages total); 15 figures, 48 tables; 213 references
Cite as BibTeX
@misc{spangher2026verifiable,
title = {Verifiable, Articulable, and Tacit Components of Preference},
author = {Spangher, Alexander and Huang, Sheldon S. and Haupt, Andreas and Goodman, Noah D. and Yang, Diyi and Ho, Daniel E. and Koyejo, Sanmi},
year = {2026},
eprint = {2610.03025},
archivePrefix = {arXiv},
primaryClass = {cs.AI},
doi = {10.48550/arxiv.2610.03025},
url = {https://arxiv.org/abs/2610.03025}
}