Security benchmarks miscount the cost of defending image models
A study showed that standard security benchmarks misread accuracy drops as backdoor removal costs, and presented a tool that measures defenses on clean twin models.
Edited by Boris Djordjevic · Paperfoot
In a preprint, the authors introduced TARE, a method tested on image-classification models in BackdoorBench, a benchmark for backdoor defenses. Backdoor attacks use hidden triggers to make models misbehave. Leaderboards measure defense costs as accuracy drops on ordinary images. For three of sixteen attacks, a training-schedule setting never activated. On the PreAct-ResNet18 image model, fine-tuning defenses improved accuracy from a low starting point, making the reported cost negative.
TARE measures what a defense does to a never-poisoned twin trained with the same recipe, schedule and seed. On the BadNets attack grid, seven of eight defenses reduced the twin’s accuracy, with reported costs from 0.13 to 5.70 percentage points. Neural Cleanse did so only when its detector activated. The eighth defense destroyed the twin.
Why it matters
The public-benchmark tests showed why an apparent accuracy gain need not mean backdoor removal carried no accuracy cost. TARE measures a defense’s effects on a matched twin, without identifying the cost attributable specifically to removal.
Caveats
BackdoorBench twins allowed at most 10 poisoned images and required attack success below 5%; the share of cost specifically due to backdoor removal remained unidentified. TARE did not change defense rankings within an attack, and the work has not yet been peer-reviewed.
In this paper
backdoor-defense clean-accuracy cost evaluation
- Methods
- TARE (this paper)TARE-Z (this paper)fine-tuning (baseline)fine-pruning (baseline)NAD (baseline)Neural Cleanse (baseline)ANP (baseline)ABL (baseline)I-BAU (baseline)FT-SAM (baseline)TSBD (baseline)CGD (baseline)
- Datasets
- CIFAR-10 (evaluated)CIFAR-100 (evaluated)BackdoorBench (evaluated)BackdoorBox (evaluated)Tiny-ImageNet (evaluated)the Pile subset (evaluated)Alpaca-52k (evaluated)MMLU (evaluated)TOFU (evaluated)
Results
- fine-tuning, tare, BadNets grid: +0.13; I-BAU, tare, BadNets grid: +5.70 points
On the BadNets grid seven of eight defenses charge the twin (Neural Cleanse only where its detector fires), +0.13 (fine-tuning) to +5.70 points (I-BAU); the eighth, ABL, destroys it.
Abstract
- TARE, tare, CosineAnnealingLR patch on WaNet: +0.78; TARE, tare, MultiStepLR schedule on BadNets: -2.08
With the two schedulers crossed over WaNet and BadNets, the undefended twins move from 91.50 to 94.18 % (WaNet) and from 93.71 to 91.09 % (BadNets), and the tare reverses with the scheduler, not the attack: -1.84\ \scriptstyle[-2.18,-1.50] \to +0.78\ \scriptstyle[+0.70,+0.87] on WaNet, +0.50\ \scriptstyle[+0.42,+0.59] \to -2.08\ \scriptstyle[-2.40,-1.76] on BadNets, twelve per-defense intervals excluding zero, 131/132 cells and 44/44 seeds (three defenses pooled) agreeing in sign (Table 2, Figure 1a; six defenses in Table 36 …
4.2 The Scheduler Block, Manipulated: the Main Result
- TARE, tare: +0.87
On BPP the tare moves from -2.01\ \scriptstyle[-2.63,-1.40] to +0.87\ \scriptstyle[+0.69,+1.06] (Input-Aware has no admissible twin, A4 0/5 ); FT-SAM flips as fine-tuning does ( -2.14\ \scriptstyle[-2.46,-1.81] \to +1.09\ \scriptstyle[+0.92,+1.26] ), SAU lands at a lower level of its own ( 91.4 %) and I-BAU has none (Figure 10); on CIFAR-100 the twin’s rise doubles ( +6.01 on WaNet, +6.75 on BPP) and an annealed control given the block flips the other way (Blended, +2.76\ \scriptstyle[+2.50,+3.03] \to -5.23\ \scriptstyle[-7.60,-2.86] ); on VGG19-BN fine-tuning flips ( -2.30\ \scriptstyle[-2.72,-1.89] \to +2.09\ \scriptstyle[+1.59,+2.59] ); and in a second toolkit, BackdoorBox, whose shipped 200 -epoch WaNet run anneals, cutting it to 100 epochs under its own milestones starts the twin 8.61 points lower and hands fine-tuning and NAD a “gain” ( -6.10, -7.23 ) that decay erases or reverses ( +0.01, +0.81 ), while pruning, which trains nothing, charges both (Appendix S33 …
4.3 The Sign Replicates; the Level Does Not Always
- FT-SAM, tare: +1.09
… FT-SAM flips as fine-tuning does ( -2.14\ \scriptstyle[-2.46,-1.81] \to +1.09\ \scriptstyle[+0.92,+1.26] ), SAU lands at a lower level of its own ( 91.4 %) and I-BAU has none (Figure 10); on CIFAR-100 the twin’s rise doubles ( +6.01 on WaNet, +6.75 on BPP) and an annealed control given the block flips the other way (Blended, +2.76\ \scriptstyle[+2.50,+3.03] \to -5.23\ \scriptstyle[-7.60,-2.86] ); on VGG19-BN fine-tuning flips ( -2.30\ \scriptstyle[-2.72,-1.89] \to +2.09\ \scriptstyle[+1.59,+2.59] ); and in a second toolkit, BackdoorBox, whose shipped 200 -epoch WaNet run anneals, cutting it to 100 epochs under its own milestones starts the twin 8.61 points lower and hands fine-tuning and NAD a “gain” ( -6.10, -7.23 ) that decay erases or reverses ( +0.01, +0.81 ), while pruning, which trains nothing, charges both (Appendix S33 …
4.3 The Sign Replicates; the Level Does Not Always
- fine-tuning, tare, on VGG19-BN: +2.09
… on VGG19-BN fine-tuning flips ( -2.30\ \scriptstyle[-2.72,-1.89] \to +2.09\ \scriptstyle[+1.59,+2.59] ); and in a second toolkit, BackdoorBox, whose shipped 200 -epoch WaNet run anneals, cutting it to 100 epochs under its own milestones starts the twin 8.61 points lower and hands fine-tuning and NAD a “gain” ( -6.10, -7.23 ) that decay erases or reverses ( +0.01, +0.81 ), while pruning, which trains nothing, charges both (Appendix S33 …
4.3 The Sign Replicates; the Level Does Not Always
- fine-pruning, points, at 1% clean data: 3.79; fine-pruning, points, at 100% clean data: 0.28
… fine-pruning costs 3.79 [3.25,4.33] points at 1% clean data and 0.28 [0.09,0.47] at 100% (Appendix S16); a cost column is not comparable across papers that omit b …
3.3 Admissibility and Scope
- TSBD, points, on CIFAR-10, on the never-poisoned twin shipped: -1.11
… on the never-poisoned twin it “gains” -1.11\ \scriptstyle[-1.86,-0.35] shipped ( 38% of the victim’s gain at its printed default; Table 59 sweeps the ratio) and costs +2.59\ \scriptstyle[+1.37,+3.82] under cosine …
4.4 What the Artifact Reached: Leaderboard, Rating, Papers
- CGD, points, on CIFAR-10, in the same 2×2 the twin shipped: -2.43
CGD reports “improving CA over the No Defense baseline by 3.4% for WaNet”; in the same 2× 2 the twin gains -2.43\ \scriptstyle[-3.00,-1.86] ( 66% of the victim’s gain) and loses +0.41\ \scriptstyle[+0.26,+0.57] under cosine.
4.4 What the Artifact Reached: Leaderboard, Rating, Papers
What it shows
- A single scheduler configuration line determines the sign of the fine-tuning family's clean-model cost on a never-poisoned twin across 131 out of 132 evaluation cells.
Edit that one lr_scheduler line and the clean-accuracy cost a defense charges a never-poisoned twin changes sign both ways (Figure 1a): 131/132 (seed, defense) cells follow the schedule, not the attack.
1 Introduction
- Backdoor defense costs measured on poisoned victims inherit the victim's starting point and cannot separate removal cost from baseline model effects.
Measured on the poisoned victim alone, the drop cannot separate removal from what the defense does to any model, and inherits the victim's start, which for three of BackdoorBench's sixteen attacks is a configuration file: WaNet, BPP and Input-Aware ship a MultiStepLR that never fires, so their victims never anneal and are the least accurate in 30/31 public CIFAR cells at ≤5 …
Abstract
- Seven of eight defenses charge a never-poisoned twin positive clean-accuracy costs on the BadNets grid, while ABL destroys the twin.
On the BadNets grid seven of eight defenses charge the twin (Neural Cleanse only where its detector fires), +0.13 (fine-tuning) to +5.70 points (I-BAU); the eighth, ABL, destroys it.
Abstract
- Within an attack, the victim's starting accuracy cancels from rankings, so the tare does not reorder defenses within that attack.
Within an attack the start cancels from rankings, so the tare re-orders nothing there; what poisoning adds beyond it is printed under two estimators and not corrected, its removal share unidentified.
Abstract
- Re-running TSBD and CGD with their released code shows that their reported clean accuracy gains reproduce on a never-poisoned model.
… TSBD and CGD, re-run with their code, "gain" on a never-poisoned model too …
Abstract
Quoted word for word from the full text on arXiv.
The paper
TARE: Weigh a Never-Poisoned Twin Before Reading Backdoor-Defense Costs
arXiv, Oct 4, 2026, Open access, CC BY, Preprint, not peer-reviewed
doi.org/10.48550/arxiv.2610.06994
arXiv:2610.06994v1 in cs.CR, also cs.AI
Comments: 84 pages (9-page main text)
Cite as BibTeX
@misc{xu2026tare,
title = {TARE: Weigh a Never-Poisoned Twin Before Reading Backdoor-Defense Costs},
author = {Xu, Ruizhi and Xu, Wei and Zhu, Sibo},
year = {2026},
eprint = {2610.06994},
archivePrefix = {arXiv},
primaryClass = {cs.CR},
doi = {10.48550/arxiv.2610.06994},
url = {https://arxiv.org/abs/2610.06994}
}
