Speech tokenizers turn out to be a semantic attack surface: T-SemAttack lifts word error rates on nine ASR systems from roughly 1–4% to 20–63.6%

Speech tokenizers are fragile at the semantic level: a small waveform perturbation can disrupt their semantic encoding and drag down downstream ASR and large audio-language model recognition. Earlier assessments of tokenizer fragility mainly measured tolerance to Gaussian noise, while transferable attacks on semantic representations and their propagation mechanisms remained less explored. On the Fluent Speech Commands task, T-SemAttack causes substantially larger downstream degradation than Gaussian noise, yielding scores even lower than those under stronger Gaussian noise at an SNR of 5 dB.

The attack pushes perturbations that move several semantic encoders' representations away from clean speech at once, rather than fitting the output of one target model. Existing speech attacks mostly rely on specific output targets of ASR or large audio-language models and require access to the overall architecture, which raises optimization cost and limits transferability. Four surrogate encoder branches — S3Tokenizer, HuBERT, the Whisper encoder and MFCC — are optimized jointly, with the perturbation bounded by a psychoacoustic masking threshold, reaching a real-time factor of 21.800 and an SNR of 16.319 dB on LibriTTS.

One perturbation transfers to unseen models: on single-speaker English LibriTTS, word error rates across nine ASR systems rise from about 1–4% on clean speech to 20.004–63.555%. For comparison, the end-to-end attack E2E-VGuard averages 8.473% word error rate on the same data, and Gaussian noise averages 3.376%. The average word error rate is 81.872% on the multi-speaker English Seed-TTS subset and 50.814% on the Chinese subset; across 11 dataset-model combinations, large audio-language models average 58.30% word error rate, above E2E-VGuard's 16.76% and AudioShield's 38.66%.

ROSETok moves the defense back into the tokenizer, using noisy training and semantic distillation so the encoder stays close to clean-speech semantic representations under noisy input. Defending only at the language-model level costs more time and compute, and existing tokenizers lose substantial downstream score under semantic attack, for example StableToken dropping from 96.7 on clean input to 35.0. On Seed-TTS English data, ROSETok-Base reaches reconstruction STOI of 0.966, 0.950 and 0.842 on clean, Gaussian-noise and T-SemAttack inputs; attack-aware fine-tuned ROSETok-AAF raises the attacked-input reconstruction score to 0.903 and the downstream FSC score from 62.4 to 77.9.

The evaluation covers single-speaker English LibriTTS, multi-speaker English and Chinese Seed-TTS, nine ASR systems and four large audio-language models; the attack runs in an offline setting with minimal real-time requirements. ROSETok-Base is trained on roughly 96.7k hours of English and Chinese Emilia data, and the second stage performs attack-aware fine-tuning on perturbations generated from 2,000 utterances randomly selected from LibriSpeech.

The authors state that T-SemAttack aims at not disrupting normal usability of the audio rather than true imperceptibility, with a subjective MOS-Q of 3.01; they also note that ROSETok is not a complete mitigation against all semantic attacks, that its robustness depends on the training distribution and threat model, and that deployed systems should combine it with monitoring, denoising and system-level safeguards. They list improving the attack's real-time performance, for example through multi-GPU parallelism, as future work.