Audit corrects 19 percent of security flaw labels in code datasets
An auditing framework tested 35,849 code instances across three datasets, confirming 57.2% of vulnerability labels and correcting 19.0% by executing flaw-triggering tests.
Edited by Boris Djordjevic · Paperfoot
The authors built VulValidate, a framework that uses large language model agents to coordinate dynamic analysis tools and construct vulnerability-triggering experiments, and used it in a preprint to audit 35,849 instances labeled vulnerable in BigVul, PrimeVul, and DiverseVul. Given a function and its fixing patch, the framework reconstructed revisions before and after the fix, refined triggering inputs, and compared runtime behavior.
The audit confirmed 20,510 labels (57.2%), corrected 6,819 (19.0%), left 7,981 attacked but undecided (22.3%), and could not successfully measure 539 (1.5%). In a blinded review of 581 sampled decisions, expert consensus supported 90.0%–92.0% of confirmations and 92.6%–99.0% of label corrections. With model parameters fixed, evaluation on corrected labels lowered F1, an accuracy score, for all five tested detectors on BigVul and DiverseVul. Retraining with corrected labels improved F1 for four of five detectors on each dataset.
Why it matters
The findings show that security datasets built from patch histories frequently mislabel harmless code changes as vulnerabilities, distorting how well automated flaw detectors perform. By anchoring labels in executable runtime evidence, the approach helps developers train and benchmark security models on verified data.
Caveats
The framework left 22.3% of audited instances undecided and could not measure 1.5%, while human validation covered a sample of 581 decisions rather than the full set. The study is also a preprint that has not yet undergone peer review.
In this paper
auditing function-level vulnerability labels
- Methods
- VulValidate (this paper)Devign (evaluated)ReVeal (evaluated)DeepDFA (evaluated)LineVul (evaluated)CodeT5 (evaluated)Claude Opus 5.0
- Datasets
- BigVul (evaluated)PrimeVul (evaluated)DiverseVul (evaluated)
- Links
- Project pageCode
Results
- LineVul, precision, on BigVul, fixed threshold of 0.5, O/O to O/C: 0.4670; LineVul, AUROC, on BigVul, fixed threshold of 0.5, O/O to O/C: 0.9599
At LineVul’s fixed threshold of 0.5 on BigVul, precision falls from 0.9074 to 0.4670, while AUROC changes from 0.9646 to 0.9599.
4.4. RQ4: Vulnerability detector performance changes after label correction
- DeepDFA, F1, on DiverseVul, O/O to C/C: 0.2217
DeepDFA is the exception on DiverseVul, improving from 0.1927 to 0.2217.
4.4. RQ4: Vulnerability detector performance changes after label correction
- CodeT5, F1 gain, on BigVul, O/C to C/C retraining: 0.0949; DeepDFA, DiverseVul gain, on DiverseVul, O/C to C/C retraining: 0.0903
CodeT5 has the largest BigVul F1 gain (0.0949), while DeepDFA has the largest DiverseVul gain (0.0903).
4.4. RQ4: Vulnerability detector performance changes after label correction
- DeepDFA, F1, on BigVul, O/E versus C/E: 0.7139
On BigVul, C/E F1 is lower than O/E for CodeT5, LineVul, Devign, and ReVeal, while DeepDFA improves from 0.6966 to 0.7139.
4.4. RQ4: Vulnerability detector performance changes after label correction
- CodeT5, recall: 0.3966; CodeT5, precision: 0.4471
The F1 gap is a recall gap (0.3966 versus 0.1688) at similar precision (0.4471 versus 0.4888).
4.5. Vulnerability detection performance on the merged dataset
- CodeT5, percentage of positive predictions that are correct: 44.71%
Accuracy above 0.96 for the transformer models reflects the 97.04% non-vulnerable majority, not detection: the best detector, CodeT5, finds 39.66% of the vulnerable functions, and 44.71% of its positive predictions are correct.
4.5. Vulnerability detection performance on the merged dataset
What it shows
- Re-evaluating models trained on original labels against corrected labels lowers F1, precision, and MCC for every tested detector across both datasets.
F1, precision, and MCC decrease for every detector on both datasets.
4.4. RQ4: Vulnerability detector performance changes after label correction
- Retraining with corrected labels improves F1 and MCC for four of five detectors on each dataset.
F1 and MCC improve for four of five detectors on each dataset.
4.4. RQ4: Vulnerability detector performance changes after label correction
- Agreement between agent decisions and independent manual labels yields Cohen's kappa values of 0.55 on BigVul and 0.50 on DiverseVul.
The agent agrees with Risse et al. on 71 of 92 BigVul functions ( \kappa=0.55 ), as closely as the two human studies agree with each other ( \kappa=0.55 ), and on 68 of 87 DiverseVul functions ( \kappa=0.50 ): moderate agreement on the scale of Landis and Koch.
4.1. External validity: agreement with independent manual labels
- On the merged benchmark, every evaluated detector leaves most vulnerable functions undetected.
The merged benchmark removes the contradictory supervision described in the Introduction; on it, CodeT5 leads on F1, MCC, and AUROC, and every detector leaves most vulnerable functions undetected.
4.5. Vulnerability detection performance on the merged dataset
Quoted word for word from the full text on arXiv.
The paper
VulValidate: Auditing Function-Level Vulnerability Labels with Executable Evidence
arXiv, Oct 4, 2026, Open access, CC0, Preprint, not peer-reviewed
doi.org/10.48550/arxiv.2610.05103
arXiv:2610.05103v1 in cs.CR, also cs.AI, cs.SE
Comments: 21 pages
Cite as BibTeX
@misc{zhang2026vulvalidate,
title = {VulValidate: Auditing Function-Level Vulnerability Labels with Executable Evidence},
author = {Zhang, Leizhen and Chen, Sheng},
year = {2026},
eprint = {2610.05103},
archivePrefix = {arXiv},
primaryClass = {cs.CR},
doi = {10.48550/arxiv.2610.05103},
url = {https://arxiv.org/abs/2610.05103}
}