Skip to main content
Search

# Cross-Request KVCache Exposure

1 related articles

Subscribe to updates
  1. LMSYS Org

    When a long-context request dies mid-transfer, its KV cache can land in someone else's output: SGLang now holds slots until writes drain

    SGLang closed a cross-request KV cache exposure in disaggregated prefill/decode serving: after a request aborts mid-transfer, RDMA writes it already launched can still land and overwrite another request's cache. Decode now keeps the aborted request's slots until every notified prefill rank acknowledges a drain or a bounded timeout fires, default-on for Mooncake and NIXL from v0.5.21, with throughput deltas within ±2% on 2×H100.

You’re all caught up