A standard document benchmark has flawed labels and duplicates
An audit found approximately 10.5% of 400,000 documents had erroneous labels, and 35.2% of the test set had duplicates or near-duplicates in the training set.
Edited by Boris Djordjevic · Paperfoot
The authors of a preprint audited data quality in RVL-CDIP, a standard benchmark of 400,000 documents used for document classification. Reported accuracy on the benchmark reached 98.8%. The audit found that approximately 10.5% of its documents had erroneous labels. In the test set, 35.2% of documents had duplicates or near-duplicates in the training set. Page identification codes alone allowed 69% classification accuracy, and a state-of-the-art model lost 11 points without them. Image-only models trained on the benchmark lost 16 to 28 points on documents from other sources. Switching the optical character recognition software, which reads text from images, changed accuracy by 4.1 to 8.3 points. Nearly 16,000 documents contained sensitive personal data, which the authors de-identified. With all defects remediated, 13 retrained models were 0.9 to 2.9 points more accurate than in the original condition, led by LayoutLMv3 at 96.5%.
Why it matters
The audit of a standard public benchmark reveals how label errors, test-set duplicates, and shortcut features can distort document classification evaluations. The corrected corpus and released data give researchers a cleaner testbed to assess model performance.
Caveats
The audit evaluates a single document benchmark and retrained only 13 of the 25 models tested, while the abstract does not report variation across runs. In addition, the study is a preprint that has not yet undergone peer review.
From the paper
- Limitation
The standard tools for duplicate detection in image datasets use perceptual hashing (pHash and related methods, as used in CleanVision and fastdup), which assigns near-identical hashes to near-identical images and is unsuitable for both of the cases above.
Quoted word for word from the full text on arXiv.
In this paper
document classification data quality audit
- Methods
- LayoutLMv3 (evaluated)Donut (evaluated)DocXClassifier (evaluated)DiT-base (evaluated)DiT-large (evaluated)DiT (evaluated)LayoutLM (evaluated)LayoutLLM (evaluated)EAML (evaluated)Bi-VLDoc (evaluated)UDOP (evaluated)Dessurt (evaluated)
- Datasets
- RVL-CDIP (evaluated)RVL-CDIP-N (this paper)RVL-CDIP-O (this paper)IIT-CDIP (training)
Results
- LayoutLMv3, accuracy, on RVL-CDIP, with all defects remediated: 96.5%
We re-benchmark 25 models, and with all defects remediated, the 13 that we retrain are 0.9 to 2.9 points more accurate than in the original condition; the most accurate of them, LayoutLMv3, is 96.5% accurate.
Abstract
- Donut, accuracy, on RVL-CDIP, when the codes are removed: 84.2%
Identification codes are a usable shortcut: a classifier that uses only features of the codes is 69% accurate, and the public Donut checkpoint is 11 points less accurate when the codes are removed (95.2 against 84.2%).
Identification codes.
What it shows
- Perceptual hashing tools are unsuitable for detecting duplicate documents in RVL-CDIP due to scanner noise, geometric distortion, and varying filled content in templates.
The standard tools for duplicate detection in image datasets use perceptual hashing (pHash and related methods, as used in CleanVision and fastdup), which assigns near-identical hashes to near-identical images and is unsuitable for both of the cases above.
6.2 Limitations of Perceptual Hashing
- Differences among top published models on RVL-CDIP are smaller than the effects of the benchmark's quantified defects, failing to demonstrate which model is more accurate.
The differences between the leading published models are smaller than these effects, so the published results on RVL-CDIP do not show which of those models is more accurate.
14 Conclusion
- The audit's duplication rate, decrease in accuracy without identification codes, and label error rate serve as lower bounds.
Three of our main results are lower bounds: the duplication rate, the decrease in accuracy when the identification codes are removed, and the label error rate.
Lower bounds and definitions.
Quoted word for word from the full text on arXiv.
The paper
Data Quality in RVL-CDIP: A Systematic Audit
arXiv, Oct 23, 2026, Open access, CC BY, Preprint, not peer-reviewed
doi.org/10.48550/arxiv.2610.32126
arXiv:2610.32126v1 in cs.CV, also cs.CL
Cite as BibTeX
@misc{larson2026data,
title = {Data Quality in RVL-CDIP: A Systematic Audit},
author = {Larson, Stefan and Leach, Kevin},
year = {2026},
eprint = {2610.32126},
archivePrefix = {arXiv},
primaryClass = {cs.CV},
doi = {10.48550/arxiv.2610.32126},
url = {https://arxiv.org/abs/2610.32126}
}