KARTIK RAMESH

The authors present FluidPD, a prefill-decode (P/D) disaggregated LLM serving system built on SGLang that offloads a bounded prefill suffix to decode workers with slack via FluidToken and switches running workers between prefill and decode roles in place via FluidRole without reloading models, improving overall SLO attainment over static SGLang by up to 94.6 percentage points on Azure-trace-derived staged and mixed workloads and by up to 88.3 percentage points over Dynamo autoscaling while using fewer GPUs.

The authors present FluidPD, a prefill-decode (P/D) disaggregated LLM serving system built on SGLang that offloads a bounded prefill suffix to decode workers with slack via FluidToken and switches running workers between prefill and decode roles in place via FluidRole without reloading models, improving overall SLO attainment over static SGLang by up to 94.6 percentage points on Azure-trace-derived staged and mixed workloads and by up to 88.3 percentage points over Dynamo autoscaling while using fewer GPUs.