RubatoCodec fixes each utterance's event budget so content no longer sets sequence length, and it decodes in a stream
Synopsis
Variable-frame-rate speech codecs allocate tokens by content, but threshold-based allocation lets content set the token count, so at a fixed setting the sequence length a speech generator must produce varies across utterances and datasets. RubatoCodec adds an exact utterance-level event budget and streaming decoding to X-Codec-2, using changes in codec features and waveform spectra to cut speech into variable-duration events, each represented by discrete codes plus an explicit duration. Controlled experiments show adaptive allocation beats fixed-rate coding in reconstruction fidelity and intelligibility at low event rates; in a matched text-to-speech comparison the representation yields lower WER and higher UTMOS and UTMOSv2 with comparable speaker similarity.
Interpretation
The length of the discrete token sequence a speech generator must produce can now be pinned by an explicit per-utterance event budget, rather than left to speech content under a threshold rule. Earlier variable-frame-rate codecs used thresholds to decide where to cut, so token counts floated with content and the sequence length facing a generator at one setting differed across utterances and datasets. Controlled experiments show that at low event rates adaptive allocation outperforms fixed-rate coding in reconstruction fidelity and intelligibility.
Segmentation does not rely on a separately trained boundary predictor; it reads changes in codec features and waveform spectra to divide speech into events of unequal duration. Each event is represented by discrete codes plus an explicit duration, so a generator can predict and decode one event at a time instead of emitting a whole utterance's tokens at once. A model trained on large-scale multilingual speech reaches competitive reconstruction quality on English, Japanese and Mandarin, and has the best intelligibility preservation among the compared codecs on the evaluated English and Japanese benchmarks at comparable bitrates.
In a matched text-to-speech comparison, the adaptive representation gives lower word error rate and higher UTMOS and UTMOSv2, with comparable speaker similarity. The speech generator also learns the codec's utterance-level event budget without an explicit generation-time rate constraint. These results come from the matched text-to-speech comparison and from the generator's learning of the event budget, supporting budgeted adaptive speech representations for both high-quality reconstruction and streaming text-to-speech generation.
Perspective
The demonstrated scope is a codec built on X-Codec-2, low-event-rate reconstruction and intelligibility comparisons in controlled experiments, multilingual reconstruction evaluation on English, Japanese and Mandarin, and a matched text-to-speech comparison; code and weights are public on GitHub and Hugging Face.
The intelligibility lead is stated for the evaluated English and Japanese benchmarks at comparable bitrates; the material gives no concrete metric values and does not say how the event budget is set across datasets.