A speaker's voice can trigger hidden backdoors in speech models
The method achieved 90.2% backdoor success on the Whisper-medium speech model with a 1.7% false-positive rate, using as little as two minutes of target-speaker audio.
Edited by Boris Djordjevic · Paperfoot
The preprint presents a two-phase fine-tuning method to embed speaker-activated backdoors into speech models, tested on Whisper-medium and Parakeet. Fine-tuning adjusts a trained model on new data, and a backdoor is a hidden behavior triggered by a specific input. The method first makes speaker identity accessible to the model, then binds that speaker to an attacker-chosen payload.
Using as little as two minutes of real audio from the target speaker, the attack achieved a 90.2% backdoor success rate on Whisper-medium with a 1.7% false-positive rate, and 97.0% success on Parakeet with a 0% false-positive rate. Compared with a one-phase baseline with the same training budget, the two-phase approach substantially improved clean performance and selectivity. When transcripts were passed to language model agents, injected instructions caused them to carry out harmful actions in up to 87.5% of cases.
Why it matters
The findings show that speech models can be compromised during fine-tuning so that a person's natural voice silently triggers hidden behaviors without altering inputs during live use. Demonstrated on two speech models and frontier language agents, the work establishes speaker identity as an invisible backdoor mechanism.
Caveats
Tests were limited to two speech models, and the abstract does not report the number of runs or the datasets used for fine-tuning. The work is also a preprint that has not yet been peer-reviewed.
From the paper
- Limitation
We evaluate five target speakers from a single corpus and one voice-cloning model, and our threat model assumes access to the fine-tuning pipeline.
Quoted word for word from the full text on arXiv.
In this paper
Speaker-Activated Backdoor
- Methods
- Speaker-Activated Backdoor (SAB) (this paper)one-phase baseline (baseline)McAdams (baseline)ASR-BN (baseline)Whisper-medium (evaluated)Parakeet (evaluated)GPT-4o (evaluated)Claude Sonnet (evaluated)Qwen2.5-Omni 3B (evaluated)Qwen2.5-Omni 7B (evaluated)Whisper-large (evaluated)
- Datasets
- SHADE-Arena (evaluated)LibriSpeech (evaluated)FLEURS (evaluated)VoxPopuli (evaluated)VoxCeleb2 (evaluated)VoxCeleb (training)
Results
- Speaker-Activated Backdoor (SAB), backdoor success, using as little as two minutes of real audio from the target speaker: 90.2%; Speaker-Activated Backdoor (SAB), backdoor success, using as little as two minutes of real audio from the target speaker: 97.0%; Speaker-Activated Backdoor (SAB), false-positive rates, using as little as two minutes of real audio from the target speaker: 1.7%; Speaker-Activated Backdoor (SAB), false-positive rates, using as little as two minutes of real audio from the target speaker: 0%
SAB achieves 90.2% and 97.0% backdoor success on Whisper-medium and Parakeet, with false-positive rates of 1.7% and 0%, respectively using as little as two minutes of real audio from the target speaker.
Abstract
- Qwen2.5-Omni 7B, BSR: 92.4%
The multimodal LLMs are also backdoored successfully, and Qwen2.5-Omni 7B achieves the highest BSR of 92.4%.
6.1 Speaker identity triggers high-success, selective backdoors in ASR models.
- Whisper-large, BSR: 99.5%; Whisper-large, FPR, on the nearest-speaker group: 7.2%
Whisper-large reaches a higher BSR than Whisper-medium (99.5% vs. 90.2%), but also a higher FPR on the nearest-speaker group (7.2% vs. 2.8%).
6.3 False positives remain low on random speakers but increase for some similar voices.
- Whisper-medium, C_{1}: 93.2± 5.0%; Whisper-medium, FPR: 1.7± 1.9%; Qwen2.5-Omni 3B, C_{1}: 77.4± 11.0%; Qwen2.5-Omni 3B, FPR: 3.3± 1.9%
Whisper-medium achieves 93.2± 5.0% C_{1} with 1.7± 1.9% FPR, while Qwen2.5-Omni 3B achieves 77.4± 11.0% C_{1} with 3.3± 1.9% FPR.
8.1 Speaker-triggered instruction injection induces harmful downstream tool actions.
- GPT-4o, conditional execution rates, With Qwen2.5-Omni 3B transcripts: 94.8± 1.8%
With Qwen2.5-Omni 3B transcripts, GPT-4o reaches 94.8± 1.8%, followed by Opus at 76.0± 4.7%, Sonnet at 70.8± 7.9%, and Haiku at 48.9± 15.4%.
8.1 Speaker-triggered instruction injection induces harmful downstream tool actions.
What it shows
- Two-phase training substantially improves selectivity and clean performance compared with a one-phase baseline with the same training budget, and is especially important for content-dependent attacks.
Compared with a one-phase baseline with the same training budget, the two-phase method substantially improves selectivity and clean performance, and is especially important for content-dependent attacks.
Abstract
- The Speaker-Activated Backdoor does not survive clean fine-tuning on Whisper-medium in any evaluated setup, bounding the attack to models deployed as shipped.
For Whisper-medium, the backdoor does not survive clean fine-tuning in any setup: VoxPopuli leaves at most 2.3% BSR, and both VoxCeleb setups reduce the backdoor to 0.0% BSR.
10.1 Clean fine-tuning suppresses Whisper backdoors.
- Two minutes of real target-speaker audio is sufficient to implant the backdoor, and increasing the amount does not notably improve backdoor success rate or word error rate.
This suggests that in this setting, two minutes is sufficient to implant the backdoor, and increasing n does not notably improve BSR or WER.
9.2 Two minutes of real target speech is sufficient to implant the backdoor.
- Voice anonymization defenses can be evaded by adaptive attacks that include anonymized samples in the fine-tuning data.
Tables 8 and 8 show that voice anonymization defenses can be evaded by adaptive attacks which include anonymized samples in the fine-tuning data.
10.2 Adaptive training restores activation against voice anonymization.
- Speech models can condition malicious behavior on who is speaking rather than only on what is said.
Overall, speech models can condition malicious behaviour on who is speaking, not only on what is said.
11.2 Limitations
Quoted word for word from the full text on arXiv.
The paper
You Are the Trigger: Speaker-Activated Backdoors
arXiv, Oct 19, 2026, Open access, Preprint, not peer-reviewed
doi.org/10.48550/arxiv.2610.27210
arXiv:2610.27210v1 in cs.LG, also cs.CR, cs.SD
Cite as BibTeX
@misc{krco2026trigger,
title = {You Are the Trigger: Speaker-Activated Backdoors},
author = {Krčo, Nataša and Dodd, Euodia and Romov, Peter and de Montjoye, Yves-Alexandre},
year = {2026},
eprint = {2610.27210},
archivePrefix = {arXiv},
primaryClass = {cs.LG},
doi = {10.48550/arxiv.2610.27210},
url = {https://arxiv.org/abs/2610.27210}
}
