vLLM flips KV transfer to push: first-token latency drops to a third at high load

In disaggregated prefill/decode inference, KV cache transfer coordination and setup no longer sit on the decode instance's critical path, so decoding can begin sooner. The earlier pull mode relayed source metadata through the router only after prefill completed, and the decode instance then allocated destination blocks, prepared the transfer, and issued an NIXL READ, all serialized after prefill and before decode. On a 1P1D deployment using Amazon EC2 P5en instances connected with EFA and Qwen3-32B, prefix caching was disabled so every request fetched the full KV cache from the prefill node, and input lengths 1k–16k were swept at 1–8 QPS; at low load and short prompts the two modes performed almost identically, and pull degraded sharply as QPS and input length grew.

Push mode puts transfer setup on a background thread on the prefill side, so the decode instance no longer spends those cycles during its latency-critical token-generation phase. Pull mode posts NIXL descriptors inline on the decode worker, synchronously within its engine loop, and that work stalls token generation. At 8 QPS and 16k input, pull mode reached about 1,744 ms TTFT while push stayed near 540 ms, roughly a threefold reduction; TPOT was close at light load, around 5.6–5.8 ms, while pull climbed to about 10.4 ms at high load and long context and push held near 7 ms.

The push connector coordinates the decode instance, the prefill instance, and a background transfer thread through a three-step protocol, so destination blocks are known before prefill finishes. Pull mode required a consumer-side identifier-discovery round trip that push mode removes from the path to first token. The decode side sends its identity and reserved block locations to the prefill instance as a NIXL notification at allocation time; the prefill side pairs requests by identifier, accounting for reissued requests, then establishes a connection and issues an NIXL WRITE directly into the decoder's registered memory, with each write carrying a completion notification that the decoder counts until all producer ranks have reported.

The demonstrated setting is a 1P1D deployment using Amazon EC2 P5en instances connected with EFA, running Qwen3-32B, with input lengths 1k–16k and request rates 1–8 QPS, and prefix caching disabled so every request retrieves the full KV cache. The feature was built and validated on AWS Trainium instance clusters connected with EFA and then contributed to vLLM as an accelerator-agnostic feature, while the reported results come from a P5en GPU cluster. Using push mode requires configuring both prefill and decode servers with NixlPushConnector; other supported configurations can run it, but performance varies with hardware, model, workload, and network configuration.

Push mode requires selecting the decode instance earlier, giving up the pull mode's flexibility to choose a decode instance after prefill for load-aware routing; pull remains the better fit for workloads needing bidirectional KV transfer or deferred decode-instance selection. At light load and short prompts the two modes perform almost identically, with gains appearing mainly at high request rates and long contexts. The authors say they are exploring bringing the background NIXL threading model to the pull-mode connector and have opened a pull request toward that.