Scaling a live vLLM service no longer means a 100-second outage: Elastic EP cuts the pause to 0.82 seconds on DeepSeek V4 Pro

vLLM's Elastic Expert Parallelism can now change its GPU count with almost no serving interruption: 0.82 seconds when DeepSeek V4 Pro scales from 16 to 20 GB200 GPUs and 1.96 seconds when it scales back to 16. Earlier options added or removed whole model replicas or restarted the engine with a different GPU count, and even the non-restarting Elastic EP stopped serving throughout reconfiguration. On the same DeepSeek V4 Pro benchmark, blocking reconfiguration paused serving for 99.62 seconds on scale-up and 56.25 seconds on scale-down; asynchronous preparation alone still paused 58.04 and 44.07 seconds, and only with CUDA graph reuse did the pauses fall to 0.82 and 1.96 seconds.

Reconfiguration is split into background preparation and a short commit, so existing workers keep serving requests with their current communication groups and CUDA graphs while background threads prepare the target topology. Preparation covers initializing joining workers, setting up target communication groups, transferring model weights, and completing compilation, kernel warmup and CUDA graph construction; the commit phase only pauses schedulers, activates the prepared topology and resumes serving. Weights move from existing workers to joining workers over NVLink or RDMA, and after commit asynchronous Expert Parallel Load Balancing redistributes experts in the background while new workers begin handling attention.

NIXL EP adds or removes expert-parallel ranks in place, reusing the same NIXL agent, buffers and connections, so captured CUDA graphs stay valid across both scale-up and scale-down. Earlier communication backends had fixed membership, so adding or removing ranks required a new instance with separate buffers and memory registrations, and switching communicators invalidated captured CUDA graphs and forced warmup and recapture. On the DeepSeek V4 Pro benchmark, even with asynchronous preparation, recapturing graphs during commit still paused serving 58 seconds on scale-up and 44 seconds on scale-down; joining workers instead mask all remote expert-parallel peers during warmup, so existing workers need not participate.

The demonstrated setting is DeepSeek V4 Pro on GB200 GPUs with tensor parallelism of 1 and data-parallel and expert-parallel sizes of 16 and 20, with weights pre-downloaded in safetensors format on shared Lustre storage and measurements taken with warm compilation caches. In an 11,391-request replay with request rate rising from about 8 to 16 per second, Elastic EP completed every request and kept rolling p95 time to first token below 5 seconds, matching the static 16-GPU deployment during preparation and the recovered 20-GPU deployment after scaling. The capabilities are available in the latest vLLM and NIXL releases, and a test case includes the server configuration and scaling API calls.

The performance numbers come from a single model at 16 and 20 GPUs; behavior on other models, parallel configurations and larger scales is not reported. Total time does not fall proportionally, remaining 92.34 seconds on scale-up and 9.32 seconds on scale-down, so preparation still dominates. The stated next step is restoring initialized process and CUDA state with Dynamo Snapshot to bring new capacity up sooner, though target-topology communication must still be prepared separately and large weights are still planned to move GPU to GPU.