Tests overstated image detectors’ gains on unfamiliar data
Across two image benchmarks, most reported gains came from recognizing data sources, not from detecting images unlike those used to train a classifier.
Edited by Boris Djordjevic · Paperfoot
The study tested methods for detecting images unlike a classifier’s training data on ImageNet and CIFAR-100, two benchmarks for sorting images into categories. The detectors scored internal signals from trained image classifiers. Researchers compared two checks: new images from the same unfamiliar dataset used to fit the detector, and images from an entirely withheld dataset.
Most reported gains did not survive the second check. Changing the size of the fitted scoring rule did not change what survived. Controlled comparisons showed that inflation depended instead on whether the detector’s inputs revealed image categories. A single adjustable constant, already tuned on a designated validation dataset, survived the stricter test. More elaborate fits reported additional gains that the stricter test did not return.
Why it matters
These tests showed that checking new images from the same source can overstate gains that carry over to unseen sources. The authors also gave a formula for predicting that inflation from the fitting data, without running the stricter test.
Caveats
The evidence concerns image classifiers on these two benchmarks; the abstract gives no numerical gains or variation across runs. The work is a preprint, not yet peer-reviewed.
From the paper
- Limitation
Every combiner we fit is a ridge discriminant or one network, so another fitting rule could place the split somewhere else.
Quoted word for word from the full text on arXiv.
In this paper
post-hoc out-of-distribution detector evaluation
- Methods
- VarRatio (this paper)Mahalanobis++ (baseline)MahaVar (baseline)
- Datasets
- ImageNet (evaluated)CIFAR-100 (evaluated)OpenOOD (evaluated)ImageNet-1k (evaluated)
Results
- VarRatio, inflation: 9.18
Widening the sorted member from 13 directions to its full 1001 adds 0.84 to its inflation, while exposing class identity at a fixed 13 adds 9.18.
5.4 Class identity is what the fit buys
- VarRatio, inflation, on ImageNet, ImageNet: 8.33 AUROC; VarRatio, inflation, on CIFAR-100, CIFAR-100: 16.83
The class aware member inflates by 8.33 AUROC on ImageNet and by 16.83 on CIFAR-100.
5.4 Class identity is what the fit buys
- VarRatio, inflation, on ImageNet, permutation invariant member, ImageNet: 3.11; VarRatio, inflation, on CIFAR-100, permutation invariant member, CIFAR-100: 6.83
The permutation invariant member inflates by 3.11 and by 6.83.
5.4 Class identity is what the fit buys
- VarRatio, gap, on ImageNet, ImageNet: 5.22; VarRatio, gap, on CIFAR-100, CIFAR-100: 10.00
The gap is 5.22 and 10.00, and the class aware member inflates more on 11 of 11 ImageNet backbones and on 6 of 6 CIFAR-100 architectures, which a sign test puts at p=0.001 and p=0.031.
5.4 Class identity is what the fit buys
- VarRatio, inflation, parameter count from 13 to 5.4 · 10^5: 1.61
The inflation rises by 1.61 over that range and every step adds less than the one before.
5.3 What a fit buys and what survives
- VarRatio, hold-out arm change, parameter count from 13 to 5.4 · 10^5: -0.34
The hold-out arm moves by -0.34, so what the parameters buy stays inside the dataset.
5.3 What a fit buys and what survives
- VarRatio, reported column gain, on ImageNet, From the zero capacity row to the linear map on the feature space: 9.78; VarRatio, held-out column gain, on ImageNet, From the zero capacity row to the linear map on the feature space: 2.52
From the zero capacity row to the linear map on the feature space the reported column climbs by 9.78 on ImageNet and the held-out column climbs by 2.52.
5.3 What a fit buys and what survives
- VarRatio, reported column gain, on CIFAR-100, From the zero capacity row to the linear map on the feature space: 13.67
On CIFAR-100 the reported column climbs by 13.67 and the held-out column falls by 6.14.
5.3 What a fit buys and what survives
What it shows
- Most of the out-of-distribution detection gain reported by the usual protocol is attributable to dataset identity rather than novelty.
Most of the gain that the usual protocol reports turns out to be dataset identity rather than novelty.
Abstract
- A single constant selected on a designated validation dataset does not inflate across benchmarks or backbones, whereas every step above it fails.
Neither benchmark shows any inflation there, and no backbone moves far in either direction.
1 Introduction
- A closed-form product of displacement length and angular factor predicts measured inflation at a 0.973 rank correlation without requiring a hold-out pass.
Their product predicts the measurement at a rank correlation of 0.973 without running the hold-out protocol at all.
7 Conclusion
- The closed form assumes a linear score under one shared covariance, describing ridge discriminants exactly and the neural network only by analogy.
The closed form of Section 5.6 assumes a linear score under one covariance shared by the in-distribution and OOD clouds, so it describes the ridge discriminants exactly and the network only by analogy, and it says what the inflation is rather than what makes one material expose more class identity than another …
6 Limitations and future work
Quoted word for word from the full text on arXiv.
The paper
Dataset Identity, Not Novelty: The Source of an Inflated OOD Detection Gain
arXiv, Oct 1, 2026, Open access, CC BY, Preprint, not peer-reviewed
doi.org/10.48550/arxiv.2610.01096
arXiv:2610.01096v1 in cs.LG, also cs.CV
Comments: 30 pages, 7 figures, 33 tables
Cite as BibTeX
@misc{lee2026dataset,
title = {Dataset Identity, Not Novelty: The Source of an Inflated OOD Detection Gain},
author = {Lee, Donghoon and Kang, Shinjin},
year = {2026},
eprint = {2610.01096},
archivePrefix = {arXiv},
primaryClass = {cs.LG},
doi = {10.48550/arxiv.2610.01096},
url = {https://arxiv.org/abs/2610.01096}
}
