No sentence cuts, no waiting: CRESiST interleaves reads and writes so unsegmented streaming translation trails oracle segmentation by only 1–2 COMET points

A simultaneous speech translation system no longer has to append each new speech chunk behind the whole historical prefix; speech chunks and target tokens share one interleaved sequence, so the KV cache stays valid and the model handles continuous audio with no sentence boundaries natively. LLM-based simultaneous translation had relied mainly on prefix-incremental generation, where every appended step invalidates the KV cache, computation grows quadratically with steps, and external segmentation such as voice activity detection is needed to cut long streams. On the English-Chinese and English-German document-level MuST-C and ACL 60/60 tests, CRESiST in the unsegmented streaming setting reaches 73.73 COMET with LongYAAL 1.14 on MuST-C en-zh and 85.59 COMET with LongYAAL 1.08 on ACL 60/60 en-de, against 75.30/1.29 and 87.23/1.00 for the same model with oracle segmentation.

The read-write timing is learned without offline word-alignment supervision: a small RNN-T Actor actively samples whether to read the next speech chunk or write the next token, and a speech-LLM Critic judges whether that trajectory is good. Earlier interleaved read-write architectures depended on static word alignments from tools such as MFA to build supervised trajectories, which poorly captures an interpreter's dynamic anticipation. The Actor works at a 400 ms speech-chunk granularity and is trained on offline parallel data; the Critic is a Qwen2.5-omni-7B backbone with Router and Refiner modules that scores trajectories with a dynamic-consistency reward and a latency reward, and reverse distillation returns binarized read-write decisions to the Actor, alternating eight Critic steps with one Actor step.

The Critic's latency reward is the expected number of consecutive writes predicted by the Router, so the model is pushed to emit longer text spans as early as possible without external latency constraints. Earlier systems mostly used fixed thresholds, wait cascades, or attention-boundary indicators to decide when translation should start. On the bidirectional Chinese-English FLEURS test, the streaming model strictly outperforms the commercial Seed LiveInterpret 2.0 API in translation quality while keeping comparable or better latency, and the curve degrades gracefully at extremely low latency requirements.

The demonstrated scope is sentence-level FLEURS and CoVoST2, document-level MuST-C and ACL 60/60 in the English-Chinese and English-German directions, and a single model covering Chinese and English inputs into six target languages; document-level evaluation uses OmniSTEval, with latency computed at word level or character level for CJK languages.

The authors note that absolute translation quality is limited by backbone capacity and offline SFT data, and that offline performance on the ACL60/60 dev set drops relative to the vanilla Qwen2.5-omni model, indicating domain distribution discrepancies and catastrophic forgetting in academic domains; the decision threshold tunes the quality-latency trade-off at sentence level but has only marginal effect in document-level continuous streaming, which likely needs context-aware dynamic penalties.