Some safety benchmarks measure training policy over capability
Across ten guardrail models, benchmark scores spanned 0 to 100 on copyright and 3.4 to 56.4 percent on over-refusal, while detection capability remained unchanged.
Edited by Boris Djordjevic · Paperfoot
The preprint contributes an empirical study of ten guardrail models, which are classifiers that filter harmful text, sharing an architecture and optimizer family across two model scales. The models were trained on roughly 53,000 prompts from an industrial deployment labeled under two operational definitions: whether text requested harmful content, and whether a moderator would block it. Before evaluating on safety benchmarks, the authors removed benchmark prompts from the training pool, where 88 percent of JailbreakBench-benign and 82 percent of OR-Bench-hard were present.
Safety benchmark scores varied sharply by label definition. On HarmBench-copyright, scores across the ten models spanned 0 to 100, while on OR-Bench-hard, over-refusal spanned 3.4 to 56.4 percent at unchanged detection capability. A benchmark score could be moved across its entire range by 0.03 percent of the parameters. Validation accuracy tracked agreement with annotator policy rather than detection capability.
Why it matters
The findings show that some safety benchmark scores reflect label definitions rather than capability, meaning evaluations may measure annotator guidelines instead of how well a system detects harm. The evidence comes from an empirical study of ten guardrail models tested across multiple safety benchmarks.
Caveats
The evaluation examined ten guardrail models trained on one dataset of roughly 53,000 prompts under two specific label definitions across two model scales. The paper is a preprint that has not yet been peer-reviewed.
From the paper
- Limitation
Variance is the main weakness: each arm is a single seed, and the seed floor of §4 comes from neighbouring recipes rather than the arms themselves, in a direction we cannot test, so the middle of Figure 2(b) is unresolved …
Quoted word for word from the full text on arXiv.
In this paper
guardrail classifier evaluation
- Methods
- E1 (this paper)E2 (this paper)E5 (this paper)E6 (this paper)E7 (this paper)E8 (this paper)E9 (this paper)E10 (this paper)Granite Guardian 3.1 - 2 B (baseline)Qwen3Guard-Gen (baseline)ModernBERT-baseModernBERT-large
- Datasets
- AdvBench (evaluated)DoNotAnswer (evaluated)HarmBench (evaluated)HarmBench-contextual (evaluated)HarmBench-copyright (evaluated)HarmBench-standard (evaluated)JailbreakBench-benign (evaluated)JailbreakBench-harmful (evaluated)OR-Bench-hard (evaluated)OR-Bench-toxic (evaluated)
Results
- E8, over-refusal: 4.5 – 6.3%; E8, over-refusal, on OR-Bench-hard: 6.4%
… all four sit at 4.5 – 6.3% over-refusal against 29.7 – 30.5% for the Qwen3Guard sizes and 65.2% for Granite Guardian, and 6.4% against 55.9% and 97.0% on OR-Bench-hard alone, though on its 25 clean prompts our jury left unclear Qwen3Guard falls to 4.0 – 16.0% against our 4.0% while Granite stays at 84.0 …
The comparison
- E8, mean catch rate, in its loose mode: 91.1
On capability-loaded benchmarks E8 is level with the 8 B guard in its loose mode ( 91.1 vs 91.8 ) and behind it in strict ( 96.7 ), at 1/55 of the parameters; that parity is E8’s alone, as our four policy-A-last 149 M arms span 81.0 – 91.1.
The comparison
- E8, mean catch rate: 91.1%; E7, mean catch rate: 89.1%; E1, mean catch rate: 81.0%
The move is close to free on capability: 91.1% mean catch rate over the six capability-loaded benchmarks for E8 and 89.1% for E7 against 88.3 – 86.8% for the data-mixing arms, with E1 the exception at 81.0% (smallest calibration set, weakest starting representation).
A rank test, because one seed per arm does not license a difference of means.
- E8, catch rate: 100.0
Swapping the distillation teacher for a policy-A one while leaving every hard label policy B (E8) gives 100.0 and 6.4%; that teacher matches the policy-B one on backbone, lineage, seed and schedule but saw half as much data (Appendix A).
5 Third audit: a benchmark score is steerable at 0.03% of the parameters
- E1, catch rate: 100.0; E1, over-refusal rate: 4.7%; E10, over-refusal rate: 3.4%
Freezing the backbone and retraining only the three linear heads ( 0.04 M of 149.05 M parameters, 0.03%, on 8,489 calibration examples: 5,489 label-disagreement cases plus 3,000 generic_safe anchors) gives 100.0 and 4.7% (E1); extending it to 53,395 examples (E7) changes little ( 100.0, 6.8% ), and running the same pass on a policy-A backbone (E10) gives 100.0 and 3.4 …
5 Third audit: a benchmark score is steerable at 0.03% of the parameters
What it shows
- Safety benchmarks stratify sharply by how much of their score is decided by the label definition rather than by the model.
Second, safety benchmarks stratify sharply by how much of their score is decided by the label definition rather than by the model: with architecture and optimiser fixed, the range across the ten arms is 0.99 on HarmBench-copyright and 0.043 on WildGuard-benign, a 23x spread.
Abstract
- A guardrail benchmark score can be shifted across its full range by retraining only 0.03% of the parameters with unchanged detection capability.
Third, a benchmark score can be moved across its entire range by 0.03% of the parameters.
Abstract
- Soft labels from a distillation teacher override hard labels in establishing the policy definition of a student guardrail.
For a distilled student the teacher's soft labels do this even when every hard label carries the other policy, so teacher provenance is an unrecorded confound in any released checkpoint.
Abstract
- Validation accuracy on a held-out split tracks policy agreement rather than harmful prompt detection capability.
Fourth, validation accuracy on the one split that grades all twenty arms tracks agreement with the annotator's policy (rho=+0.74), not detection capability (rho=+0.34).
Abstract
- Over-refusal benchmarks used to measure safe deployment can be heavily contaminated in the guardrail's own training data.
First, the guardrail's own training corpus already contained the benchmarks: JailbreakBench-benign is 88% and OR-Bench-hard 82% present in the training pools, two over-refusal benchmarks, the kind a guardrail reports to argue it does not over-block.
Abstract
Quoted word for word from the full text on arXiv.
The paper
Policy Is Not Capability: Some Safety Benchmarks Measure Which Definition of "Unsafe" You Trained On
arXiv, Oct 18, 2026, Open access, CC BY-NC-ND, Preprint, not peer-reviewed
doi.org/10.48550/arxiv.2610.25060
arXiv:2610.25060v1 in cs.LG, also cs.AI, cs.CL, cs.CR
Comments: Accepted as a poster at the NeurIPS 2026 Workshop TAE (Trust-AI-Eval): Can We Trust AI Evaluation?, Sydney. 20 pages, 5 figures
Cite as BibTeX
@misc{jebraeeli2026policy,
title = {Policy Is Not Capability: Some Safety Benchmarks Measure Which Definition of "Unsafe" You Trained On},
author = {Jebraeeli, Vahid and Romero, Jonathon and Wang, Yijin and Chan, Adrian and Zhang, Jianbang},
year = {2026},
eprint = {2610.25060},
archivePrefix = {arXiv},
primaryClass = {cs.LG},
doi = {10.48550/arxiv.2610.25060},
url = {https://arxiv.org/abs/2610.25060}
}