Multi-turn agents stop recomputing history: reverse KV transfer cuts time to first token by up to 3.3x

In multi-turn conversations, the prefill node can now take KV cache left behind by the decode node in the previous turn instead of recomputing the whole history. Conventional disaggregated serving moved KV cache only from prefill to decode, so the decode node's cache for the prompt and generated tokens could not flow back, and the next turn forced the prefill node to recompute KV projections for the previous response tokens. On a 1P1D deployment with prefill prefix caching disabled to emulate an evicted cache, enabling the feature changed Qwen3-32B time to first token from 158 ms at 2k rising to 778 ms at 24k into 122 ms rising to 460 ms, and GLM-5-FP8 from 500 ms at 2k rising to 2,334 ms at 20k into 358 ms rising to 702 ms.

The mechanism keeps KV blocks alive past the turn that produced them and moves them in the reverse direction, coordinated by a KV-cache-aware router. The NVIDIA Inference Xfer Library KV connector was extended so prefill nodes can consume remote KV blocks much as decode nodes already do, with both directions sharing the same NIXL READ transfer mechanism. The router keeps only an in-memory mapping from conversation_id to the decode node's kv_transfer_params block locations, not KV tensors, and reads that metadata from the response stream; the decode node retains blocks for decoder_kv_blocks_ttl, default 480 seconds, and the prefill node queues an asynchronous receive once reusable tokens reach kv_recompute_threshold, default 64.

Results come from a 1P1D deployment, one prefill node and one decode node, using Amazon EC2 P5en instances connected with EFA, on Qwen3-32B and zai-org/GLM-5-FP8. The benchmark disabled prefix caching on the prefill node to emulate a locally evicted cache or a router choosing a different prefill node without the conversation's KV blocks. The feature was contributed to vLLM as an accelerator-agnostic capability, built and validated on EFA-connected AWS Trainium clusters, and the results presented here come from a P5en GPU cluster, showing the benefits also carry over to GPUs. Implementing it requires vLLM v0.21.0 or later with NixlConnector, bidirectional_kv_xfer enabled on both prefill and decode servers, and a KV-cache-aware router.

Performance data depend on workload and configuration, and the reported results vary with hardware, model, workload and network configuration. The block retention time decoder_kv_blocks_ttl is fixed at 480 seconds and not renewed, so it must be tuned to the expected time between turns to balance KV reuse against accelerator memory; turns arriving later fall back to normal prefill. KV reuse is not always faster than local recomputation, since for a short reusable prefix interconnect and coordination overhead can exceed recompute time. Extending the heartbeat-renewed lease to the decode side is described as a natural next improvement.