SGLang grows a live MoE deployment instead of restarting it, reaching the first request 5.3× to 8.2× faster
Synopsis
Large mixture-of-experts (MoE) inference deployments previously had two scaling options: add a full model replica, or restart into a wider expert-parallel topology. SGLang's elastic expert-parallel scale-up adds GPU ranks to a running deployment, reserving metadata and communication buffers for max_ep_size at launch, admitting joiners through Mooncake PG, and carrying expert traffic over NIXL before committing the new topology. On 8×H100 with DeepSeek-V3-Lite FP8, scale-up reached the first completed request 5.3× to 8.2× faster than restart.
Interpretation
A running MoE inference deployment can now add GPU ranks in fine-grained increments rather than allocating another full model replica or restarting the job. Horizontal scaling added the full model footprint, while vertical scaling required rebuilding distributed state, reloading the model, and warming the new deployment. In a 20→24 GPU example, four ranks join the serving deployment instead of a second 20-GPU replica or a replacement job.
Scale-up reuses the membership machinery of the failure-recovery flow, turning an unexpected membership change into a planned one. Recovery returns a rank to its existing slot and original process groups, whereas scale-up activates unused tail slots within the max_ep_size ceiling. Launch-time process groups are static, so the original ranks and each joining group keep fixed tensor-parallel and bootstrap groups while Mooncake WORLD and logical EP membership grow above them.
Three layers cooperate: SGLang owns the serving control plane, Mooncake PG handles elastic membership and synchronization, and NIXL carries the expert data plane. SGLang tracks pending_ep_size separately from effective_ep_size, installs target expert and DP state at the end of a forward, and exposes new scheduler slots only after commit. On EP4→EP6 the original ranks keep existing connections and add ranks 4 and 5, joiners connect to the EP6 peer set, and NIXL metadata and RDMA buffers are configured for max_ep_size up front while connections complete lazily.
Perspective
The demonstrated setting is DeepSeek-V3-Lite FP8 on 8×H100, compared against restart, with timing from the scale or restart request to the first completed request on the target topology, including first-use connection setup. The intended reader is an engineer running large MoE inference services; the scale API admits processes but does not allocate GPUs, so an external orchestrator must launch joining ranks. max_ep_size should be set to the largest topology the deployment is expected to reach, and a higher ceiling reserves more communication-buffer memory on active GPUs, leaving less for the KV cache.
Automatic rollback is not yet supported once joining ranks enter Mooncake WORLD, and a failure during finalization can require restarting the deployment. Token delivery pauses during the topology transition. CUDA graph recapture currently covers full decode only, with prefill and piecewise recapture unsupported. Recovery after scale-up, scale-down and grow-back, and fully idle progress remain under review or planned.