Picking one speaker out of overlapping speech: a 5.2M frontend spends compute by difficulty, cutting WER from 16.8% to 14.6%
Synopsis
Target speaker extraction for multilingual overlapping speech no longer applies one fixed computation depth to every input. Fixed-depth frontends either under-process hard inputs or waste computation on inputs that have already converged. LATTE uses a difficulty estimator to route each mixture through two, four, or six shared extractor blocks, reaching 15.07 dB macro SI-SNRi at 0.30 GFLOPs on the eight-language PolyExtract benchmark, within 0.09 dB of a 0.53-GFLOP fixed-depth path while cutting computation by 43.4%; before frozen Qwen3-Omni it lowers average WER from TF-GridNet's 16.8% to 14.6%.
Interpretation
A target speaker extraction frontend now allocates computation depth per mixture instead of running the same network depth on every input. Earlier extraction systems applied uniform computation, so a shallow budget under-processed demanding inputs while a deep budget repeatedly refined inputs that had already converged. On PolyExtract, covering eight languages and 64 directed target-interferer conditions, LATTE reaches 15.07 dB macro SI-SNRi at 0.30 GFLOPs, within 0.09 dB of a 0.53-GFLOP fixed-depth model while reducing computation by 43.4%.
Routing is driven by mixture features and the target-speaker embedding, so inference needs no language identity. Training uses weak supervision from a phonological difficulty index derived from multi-depth extraction gains, smoothed over a URIEL+ and PHOIBLE phonological similarity graph and used only during training. The difficulty estimator reaches a language-macro MAE of 0.085, and learned, task-PDI-only, random, and oracle routing obtain 14.69, 14.88, 13.91, and 15.23 dB, leaving a 0.16-dB gap to oracle.
The extracted waveform feeds a frozen audio language model without changing backend parameters. Backend adaptation previously relied on methods such as LoRA that are specific to a given backend, whereas this frontend stays plug-and-play. Before frozen Qwen3-Omni, average WER drops from TF-GridNet's 16.8% to 14.6%, ranking first on five of eight languages while requiring only 0.2T-0.4T FLOPs.
Perspective
The result is validated on PolyExtract, a controlled two-speaker mixture benchmark covering eight languages from five families and 64 directed target-interferer pairs, drawn from LibriSpeech, AISHELL-1, NCHLT, VIVOS, and Common Voice. The frontend outputs waveforms to a frozen audio language model, so it fits settings that want better overlapping-speech recognition without updating backend parameters. Generalization to broader language, accent, and real-world overlap conditions remains unverified, and future work should evaluate unseen languages, accents, and real-world overlap together with deployment latency and energy.
The phonological difficulty index is explicitly framed as a training prior rather than a deterministic or intrinsic measure of language difficulty, and it only smooths task-observed evidence and is not accessed at test time. PolyExtract covers only eight languages and controlled two-speaker mixtures, so generalization to broader language, accent, and real-world overlap conditions remains unverified. On downstream semantic understanding, the frontend improves over mixture input but still trails the clean single-speaker reference on tasks such as slot filling, intent inference, and emotion recognition.