# KV Cache Transfer

BridgeKV lets a large model in a model family run prefill once, then a lightweight KV-stitching module translates the large model's prompt cache into the small model's native cache layout, so every output token is generated by the small model alone. Across six model pairs and five datasets, it scores above fine-tuned small-model baselines with matched trainable parameter budgets in all 30 settings, closing up to 80% of the quality gap to running the large model for both phases. On one A40, Llama-3.1-Instruct-8B→Llama-3.2-Instruct-1B serves a 1K-token prompt with 256 output tokens in 2.96 s, close to the 2.80 s of the small model alone and 3.0× faster than the 8.83 s of the large model alone.

BridgeKV lets a large model in a model family run prefill once, then a lightweight KV-stitching module translates the large model's prompt cache into the small model's native cache layout, so every output token is generated by the small model alone. Across six model pairs and five datasets, it scores above fine-tuned small-model baselines with matched trainable parameter budgets in all 30 settings, closing up to 80% of the quality gap to running the large model for both phases. On one A40, Llama-3.1-Instruct-8B→Llama-3.2-Instruct-1B serves a 1K-token prompt with 256 output tokens in 2.96 s, close to the 2.80 s of the small model alone and 3.0× faster than the 8.83 s of the large model alone.

The SGLang team, with Google and Meta, released SGLang-TorchTPU, an open-source backend that runs the same SGLang serving engine on both GPUs and TPUs. TPU had not been a native PyTorch device, so a PyTorch-centric serving stack could not be reused directly; the new backend keeps the scheduler, RadixAttention and API server unchanged and places TPU work underneath. It is already running in production at Meta and serves as the rollout engine for reinforcement learning with TorchTitan through Miles.

The SGLang team, with Google and Meta, released SGLang-TorchTPU, an open-source backend that runs the same SGLang serving engine on both GPUs and TPUs. TPU had not been a native PyTorch device, so a PyTorch-centric serving stack could not be reused directly; the new backend keeps the scheduler, RadixAttention and API server unchanged and places TPU work underneath. It is already running in production at Meta and serves as the rollout engine for reinforcement learning with TorchTitan through Miles.

The authors present FluidPD, a prefill-decode (P/D) disaggregated LLM serving system built on SGLang that offloads a bounded prefill suffix to decode workers with slack via FluidToken and switches running workers between prefill and decode roles in place via FluidRole without reloading models, improving overall SLO attainment over static SGLang by up to 94.6 percentage points on Azure-trace-derived staged and mixed workloads and by up to 88.3 percentage points over Dynamo autoscaling while using fewer GPUs.

The authors present FluidPD, a prefill-decode (P/D) disaggregated LLM serving system built on SGLang that offloads a bounded prefill suffix to decode workers with slack via FluidToken and switches running workers between prefill and decode roles in place via FluidRole without reloading models, improving overall SLO attainment over static SGLang by up to 94.6 percentage points on Azure-trace-derived staged and mixed workloads and by up to 88.3 percentage points over Dynamo autoscaling while using fewer GPUs.