# Disaggregated Serving

The SGLang team, with Google and Meta, released SGLang-TorchTPU, an open-source backend that runs the same SGLang serving engine on both GPUs and TPUs. TPU had not been a native PyTorch device, so a PyTorch-centric serving stack could not be reused directly; the new backend keeps the scheduler, RadixAttention and API server unchanged and places TPU work underneath. It is already running in production at Meta and serves as the rollout engine for reinforcement learning with TorchTitan through Miles.

The SGLang team, with Google and Meta, released SGLang-TorchTPU, an open-source backend that runs the same SGLang serving engine on both GPUs and TPUs. TPU had not been a native PyTorch device, so a PyTorch-centric serving stack could not be reused directly; the new backend keeps the scheduler, RadixAttention and API server unchanged and places TPU work underneath. It is already running in production at Meta and serves as the rollout engine for reinforcement learning with TorchTitan through Miles.

Multi-turn agentic conversations resend their history every turn, and KV cache previously flowed only one way, from prefill to decode, so the decode node's generated cache could not be reused. vLLM adds bi-directional KV transfer, letting the decode node retain and return computed KV blocks while the prefill node decides by a configurable threshold whether to reuse or recompute. On a 1P1D deployment of Amazon EC2 P5en instances with EFA, time to first token fell by up to 1.6x for Qwen3-32B and up to 3.3x for GLM-5-FP8 at the longest prompts, with gains growing by prompt length and depending on workload and configuration.

Multi-turn agentic conversations resend their history every turn, and KV cache previously flowed only one way, from prefill to decode, so the decode node's generated cache could not be reused. vLLM adds bi-directional KV transfer, letting the decode node retain and return computed KV blocks while the prefill node decides by a configurable threshold whether to reuse or recompute. On a 1P1D deployment of Amazon EC2 P5en instances with EFA, time to first token fell by up to 1.6x for Qwen3-32B and up to 3.3x for GLM-5-FP8 at the longest prompts, with gains growing by prompt length and depending on workload and configuration.

The vLLM-Omni team presents an orchestrator-centered multi-stage serving runtime that splits omni, TTS, diffusion, world-model and VLA workloads into stages bound to individual engines and resource budgets, moves hidden states, codec codes, latents and paged KV through OmniConnector, and adds session control for long-lived interaction; it reports H100/H200 measurements for Qwen3-Omni speech serving, a TTS portfolio, image and video diffusion, and MiniCPM-o duplex health metrics.

The vLLM-Omni team presents an orchestrator-centered multi-stage serving runtime that splits omni, TTS, diffusion, world-model and VLA workloads into stages bound to individual engines and resource budgets, moves hidden states, codec codes, latents and paged KV through OmniConnector, and adds session control for long-lived interaction; it reports H100/H200 measurements for Qwen3-Omni speech serving, a TTS portfolio, image and video diffusion, and MiniCPM-o duplex health metrics.

The authors present FluidPD, a prefill-decode (P/D) disaggregated LLM serving system built on SGLang that offloads a bounded prefill suffix to decode workers with slack via FluidToken and switches running workers between prefill and decode roles in place via FluidRole without reloading models, improving overall SLO attainment over static SGLang by up to 94.6 percentage points on Azure-trace-derived staged and mixed workloads and by up to 88.3 percentage points over Dynamo autoscaling while using fewer GPUs.

The authors present FluidPD, a prefill-decode (P/D) disaggregated LLM serving system built on SGLang that offloads a bounded prefill suffix to decode workers with slack via FluidToken and switches running workers between prefill and decode roles in place via FluidRole without reloading models, improving overall SLO attainment over static SGLang by up to 94.6 percentage points on Azure-trace-derived staged and mixed workloads and by up to 88.3 percentage points over Dynamo autoscaling while using fewer GPUs.