Voice agents stop waiting for the full sentence: Gradium hits 214 ms to first audio and 2–3x faster cold starts

A real-time voice agent no longer has to wait for the upstream language model to finish a whole sentence before it can speak; text-to-speech can start while the text is still arriving. The prior condition was to wait for full sentence generation before synthesis began, and that wait breaks the natural rhythm of a conversation. The independent Coval benchmark as of September 2026 places Gradium TTS in the faster-than-human-speech tier, with a median time-to-first-audio of 214 ms and an interquartile range tight enough for developers to set reliable latency budgets across the rest of their pipeline.

Text and audio are treated as two parallel streams on a shared clock, with the audio stream deliberately running a few decoding steps behind the text so it glimpses what comes next. Delayed Streams Modeling was developed by Kyutai; both streams align to the same frame rate of one step every 80 ms, and a longer delay gives more context and better voice quality while a shorter one lowers latency. Each audio step is split between two transformers: the larger temporal transformer runs once per step to track context and produce a vector for the next frame, while the smaller depth transformer runs up to 32 times per step to refine that vector into an audio frame and never looks back in time, keeping it cheap.

For small models the bottleneck moves from the GPU to CPU dispatch overhead, and Gradium applies two GPU optimizations so each launch does more work. In large language models the GPU is the bottleneck because kernels run long enough for the CPU to queue the next launch; in small streaming TTS models kernels finish faster than the CPU can queue them, making the CPU the latency bottleneck. Kernel fusion uses torch.compile and Triton to merge operations that previously needed several separate kernel launches into one, and CUDA Graphs record the full kernel sequence for a forward step so each later step issues a single replay command. In Gradium's published benchmarks, enabling CUDA Graphs cut median TTFA by more than 2.5x at small batch sizes.

When voice AI traffic spikes, a new replica cannot serve calls until its weights are loaded into GPU memory, and the Baseten Delivery Network shortens that cold start. Weights are typically pulled from upstream sources such as Hugging Face or S3, where rate limits and cross-region latency stretch cold starts further, especially when many replicas request the same files at once. BDN mirrors weights into Baseten-managed storage at deploy time with a content-hashed manifest, then checks node-local NVMe, an in-cluster peer cache and the mirrored origin in turn, delivering weights to H100 nodes at over 2 GB/s; during a burst each chunk is pulled from origin by one node and shared with peers, so origin bandwidth for a 50-replica scale-up is 1x the model size rather than 50x, giving 2–3x faster cold starts.

The report covers Gradium TTS as a dedicated deployment on Baseten for real-time speech synthesis, aimed at developers and engineers building real-time voice agents. The checkable quantitative scope includes a median TTFA of 214 ms on the Coval benchmark as of September 2026, a more than 2.5x cut in median TTFA at small batch sizes with CUDA Graphs enabled, and 2–3x faster cold starts from BDN.

Latency and voice quality trade off: in Delayed Streams Modeling a longer delay gives more context and better voice quality while a shorter one lowers latency, so the setting needs tuning per use case. The Coval benchmark result is stated as of September 2026, and the more than 2.5x CUDA Graphs reduction comes from Gradium's published benchmarks and is limited to small batch sizes.