Speech decoding stops waiting token by token: a draft module reuses the target model's own features for 1.9x lossless speedup on Whisper
Lead
Speech recognition decoding no longer has to wait token by token: a lightweight draft module that reuses the target model's acoustic and decoder features delivers up to 1.9x and 2.4x lossless speedups on Whisper and Conformer.
Story
Autoregressive speech recognition emits output tokens one at a time, which makes the decoding stage a latency bottleneck; EAGLE-ASR speeds up that stage without changing the recognition result. Earlier speculative decoding for ASR either depended on an external draft model strictly aligned with the target, or produced low-quality drafts whose poor acceptance rates capped the achievable speedup. On LibriSpeech, the Whisper-large-v3 target reaches 1.91x on test-clean and 1.79x on test-other with a WER change of 0.00; the Conformer AED target reaches 2.43x and 2.30x, also at 0.00.
The draft module does not stand up a separate model; it reads the target model's encoder acoustic features and multi-level decoder representations to predict upcoming tokens. The usual route trains or selects a standalone draft model, whereas EAGLE-ASR attaches a lightweight autoregressive module, Speech-Draft, to the frozen target model. Speech-Draft consists of two fully connected layers, an embedding layer, a single Transformer decoder block and an LM head, with the embedding layer and LM head reusing target-model parameters; mean accepted length is 3.67 on Whisper and 4.74 on the Conformer AED model.
What to watch
The reported scope is decoding-stage acceleration under standard full-utterance inference, not streaming recognition; the target models are Whisper-large-v3 and the FunASR Conformer AED, training and evaluation use LibriSpeech and VoxPopuli, and evaluation ran on a single NVIDIA RTX 4090 GPU with 24 GB of memory. Readers interested in multilingual and cross-lingual settings also get results for five languages and four translation directions.
The material states that from the second drafting step onward the draft module must rely on its own predicted hidden states rather than the target model's ground-truth hidden states, and this training-inference mismatch accumulates error; multi-step generation training and exponential decay weighting are introduced to mitigate it. On speech translation the mean accepted length is lower than on multilingual recognition, which the material attributes to the higher output uncertainty of cross-lingual mapping.