Speech LLMs stop pooling at a fixed frame rate: a learned boundary policy cuts English audio tokens by about a quarter

The audio prefix of a speech LLM can be split into content-adaptive segments and pooled, instead of merging every fixed number of frames. Earlier systems used fixed-rate frame stacking or pooling, whose boundaries are evenly spaced, or CTC compression, which needs a suitable auxiliary frame-level alignment target. On LibriSpeech-960h, the Transformer ASSET reaches 2.79/5.63% WER at 10.9/10.6 audio tokens per second, approaching the character-boundary oracle that uses transcripts while using fewer audio tokens.

Boundary decisions are discrete and block ordinary backpropagation, so the whole segmentation is scored by task feedback. A pooling function can be trained by backpropagation once segment assignments are given, but moving a boundary changes which frames belong to each segment. In the LibriSpeech-100h comparison, with the same Transformer policy, joint GRPO training improves on the frozen policy by 0.14/0.41 WER while using about 3 fewer tokens per second.

Each segment is summarized into one continuous audio token by a residual pooler that keeps ordering information. A segment mean is permutation-invariant within the segment and does not preserve the ordering information in the encoder features. In a comparison that holds boundaries identical and changes only the pooling method, BiGRU residual pooling improves test-clean WER by 0.25 points, with test-other nearly unchanged.

The report is for researchers and engineers working on speech LLMs and audio-token compression, covering English LibriSpeech and Mandarin AISHELL-1 recognition experiments plus exploratory adaptations to emotion recognition and speaker counting. The segmentation policy is cold-started from character-level CTC boundaries in English and phone boundaries in Mandarin, then updated by GRPO from task feedback. The non-ASR results are described by the authors as exploratory observations that the policy varies with task and with how an utterance is spoken.

The authors call the non-ASR results exploratory, reporting only that learned segmentation changes across tasks and across different expressions of the same text, without full performance conclusions for those tasks. The Mandarin experiments initialize from phone boundaries, and the authors note that Mandarin characters typically represent syllables, so phone alignment yields a higher token rate than character alignment. The policy's conditional decisions depend on previous boundary decisions, making sampling causal rather than independent.