Speech quality predictors differ widely in their hackability
Under the corrected protocol, the speech quality predictor NISQA was hacked on 90% of utterances, compared with 6% for UTMOS.
Edited by Boris Djordjevic · Paperfoot
The preprint introduced a corrected protocol and benchmark for testing four published speech quality predictors, which score audio, and audited an attack-detect-patch loop. The protocol compared attacked audio with unperturbed audio passed through the same neural codec, a model that compresses audio. This avoided charging the codec’s own score shifts to the attack. It changed measured hackability by up to a factor of four, from 0.31 to 0.08 for one defense.
Five attackers differing only in random seed, a setting controlling randomness during training, reached success rates from 0.00 to 0.38 against one fixed predictor. Under the corrected protocol, NISQA was hacked on 90% of utterances, against 21% for SSL-MOS. DNSMOS was hacked on 14%, and UTMOS on 6%. Patching helped only against attacks within the loop’s own range; a baseline using random changes matched it.
Why it matters
Speech quality predictors are increasingly used as training rewards. These tests showed that accounting for audio processing and testing several attackers mattered when assessing the four predictors’ resistance to attacks.
Caveats
Patching’s benefit was smaller than the spread between attackers, and it cost up to 0.30 in system-level rank correlation—a measure of ranking agreement—outside its original domain. The abstract does not report the benchmark’s speech datasets or languages, and the paper is a preprint, not yet peer-reviewed.
In this paper
measuring hackability of speech quality predictors
- Methods
- random-perturbation baseline (baseline)NISQA (evaluated)SSL-MOS (evaluated)DNSMOS (evaluated)UTMOS (evaluated)Enhancers (evaluated)neural codec
Results
- NISQA, percentage of utterances hacked: 90%; SSL-MOS, percentage of utterances hacked: 21%; DNSMOS, percentage of utterances hacked: 14%; UTMOS, percentage of utterances hacked: 6%
Under this protocol, four published predictors differ widely: NISQA is hacked on 90% of utterances, SSL-MOS on 21%, DNSMOS on 14% and UTMOS on 6%.
Abstract
- Enhancers, PESQ, post-trained against patched predictors: -0.03
Enhancers post-trained against patched predictors hack them far less (PESQ -0.03 versus -0.23).
Abstract
What it shows
- Patched predictors are hardened only in their own attack space by less than the spread between attackers, matched by a random-perturbation baseline, and cost up to 0.30 system SRCC out of domain.
It hardens the predictor only in its own attack space, by less than the spread between attackers; a random-perturbation baseline matches it; and it costs up to 0.30 system SRCC out of domain.
Abstract
- Referencing the unperturbed round trip instead of scoring against raw input changes measured hackability by up to a factor of four.
Referencing the unperturbed round trip instead changes measured hackability by up to a factor of four (0.31 to 0.08 for one defence).
Abstract
- Five attackers differing only in random seed obtain success rates from 0.00 to 0.38 against a single fixed predictor.
Second, one trained attacker is a sample, not a measurement: five attackers differing only in random seed reach success rates from 0.00 to 0.38 against one fixed predictor, so a defence claim needs the worst case over several.
Abstract
Quoted word for word from the abstract on arXiv.
The paper
arXiv, Oct 7, 2026, Open access, CC BY, Preprint, not peer-reviewed
doi.org/10.48550/arxiv.2610.10899
arXiv:2610.10899v1 in cs.LG
Cite as BibTeX
@misc{alavi2026hackable,
title = {How Hackable Is Your Speech Quality Metric? A Corrected Protocol, a Benchmark, and What Patching Buys},
author = {Alavi, Ali and Williamson, Donald S.},
year = {2026},
eprint = {2610.10899},
archivePrefix = {arXiv},
primaryClass = {cs.LG},
doi = {10.48550/arxiv.2610.10899},
url = {https://arxiv.org/abs/2610.10899}
}