AI4S News · AI and science frontiers

When a long-context request dies mid-transfer, its KV cache can land in someone else's output: SGLang now holds slots until writes drain

SGLang closed a cross-request KV cache exposure in disaggregated prefill/decode serving: after a request aborts mid-transfer, RDMA writes it already launched can still land and overwrite another request's cache. Decode now keeps the aborted request's slots until every notified prefill rank acknowledges a drain or a bounded timeout fires, default-on for Mooncake and NIXL from v0.5.21, with throughput deltas within ±2% on 2×H100.

vLLM Blog Oct 20, 2026

vLLM models now serve on Google TPUs without code changes, as a new backend reuses the existing scheduler and KV cache

Inferact and Google released vLLM TorchTPU, a backend that serves model definitions already in vLLM on Google Cloud TPUs without changing model code. Previously, bringing a model to TPU meant writing separate model code; this backend keeps vLLM's command line, OpenAI-compatible server, scheduler and KV cache manager, with TPU-specific work sitting underneath them. It compiles through torch.compile's TPU backend to XLA and uses custom JAX/Pallas kernels for attention, mixture-of-experts and quantized matmul.

vLLM Google

vLLM TorchTPU Pallas Kernels Speculative Decoding Disaggregated Prefill/Decode Serving KV Cache