# SGLang

Muse Spark Open decoding rises from a 517 tok/s baseline to 1,109 tok/s by fusing each layer's attention and MoE into a single kernel that stays resident on the GPU, overlapping weight loads, tensor-core work and cross-GPU communication. Previously SGLang launched about 1,450 kernels per decoding step, and the GPU drained and refilled between them, leaving some SMs and threads idle. The figure was measured on Blackwell GPUs at batch size one with DFlash V2 speculative decoding and has not yet been extended to larger batches.

Muse Spark Open decoding rises from a 517 tok/s baseline to 1,109 tok/s by fusing each layer's attention and MoE into a single kernel that stays resident on the GPU, overlapping weight loads, tensor-core work and cross-GPU communication. Previously SGLang launched about 1,450 kernels per decoding step, and the GPU drained and refilled between them, leaving some SMs and threads idle. The figure was measured on Blackwell GPUs at batch size one with DFlash V2 speculative decoding and has not yet been extended to larger batches.

For decision tasks such as checking an order's status, an application often needs a category or a score rather than prose. SGLang's Score API lets callers declare label tokens explicitly and read their scores directly, while multi-item scoring (MIS) reuses the repeated query computation inside one request and keeps candidates isolated. On Open-Jev choice tasks, MIS latency stays nearly flat as options grow from 2 to 16: 18.7 ms on Qwen3-0.6B versus 39.6 ms for Generate and 24.3 ms for SIS; the advantage is largest as load rises and is not consistently faster at low load.

For decision tasks such as checking an order's status, an application often needs a category or a score rather than prose. SGLang's Score API lets callers declare label tokens explicitly and read their scores directly, while multi-item scoring (MIS) reuses the repeated query computation inside one request and keeps candidates isolated. On Open-Jev choice tasks, MIS latency stays nearly flat as options grow from 2 to 16: 18.7 ms on Qwen3-0.6B versus 39.6 ms for Generate and 24.3 ms for SIS; the advantage is largest as load rises and is not consistently faster at low load.

SGLang adds NVFP4 storage to its paged KV cache, cutting per-token key-value bytes to about 56% of FP8. On Qwen3.8-27B at 32K, 160K and 1M input lengths with matched decode concurrency, peak-batch decode throughput rises about 26–30%; on agentic load beyond concurrency 12, FP8 throughput collapses while NVFP4 keeps climbing. Accuracy gaps vary by task, reaching 1.60 points on SWE-bench Verified.

SGLang adds NVFP4 storage to its paged KV cache, cutting per-token key-value bytes to about 56% of FP8. On Qwen3.8-27B at 32K, 160K and 1M input lengths with matched decode concurrency, peak-batch decode throughput rises about 26–30%; on agentic load beyond concurrency 12, FP8 throughput collapses while NVFP4 keeps climbing. Accuracy gaps vary by task, reaching 1.60 points on SWE-bench Verified.

DeepSeek open-sourced DeepInfer, the inference engine behind its own service, along with its kernels; SGLang then brought the Mega kernel series, DSpark speculative decoding, SWA bounded replay and both Engram placements into its mainline and compared against DeepInfer on the same hardware using DeepInfer's own benchmarks. On 4x GB300 and 4x B300, SGLang reaches up to 1.04x DeepInfer's prefill throughput, within 1% at the remaining points; on 64x GB300 its decode step is 1.1% to 7.4% shorter from 16 to 256 requests per GPU and on par at 8. SGLang also ships the KV cache system, prefill/decode disaggregation and cross-replica load balancing that DeepInfer's release notes list as not included.

DeepSeek open-sourced DeepInfer, the inference engine behind its own service, along with its kernels; SGLang then brought the Mega kernel series, DSpark speculative decoding, SWA bounded replay and both Engram placements into its mainline and compared against DeepInfer on the same hardware using DeepInfer's own benchmarks. On 4x GB300 and 4x B300, SGLang reaches up to 1.04x DeepInfer's prefill throughput, within 1% at the remaining points; on 64x GB300 its decode step is 1.1% to 7.4% shorter from 16 to 256 requests per GPU and on par at 8. SGLang also ships the KV cache system, prefill/decode disaggregation and cross-replica load balancing that DeepInfer's release notes list as not included.

Large mixture-of-experts (MoE) inference deployments previously had two scaling options: add a full model replica, or restart into a wider expert-parallel topology. SGLang's elastic expert-parallel scale-up adds GPU ranks to a running deployment, reserving metadata and communication buffers for max_ep_size at launch, admitting joiners through Mooncake PG, and carrying expert traffic over NIXL before committing the new topology. On 8×H100 with DeepSeek-V3-Lite FP8, scale-up reached the first completed request 5.3× to 8.2× faster than restart.

Large mixture-of-experts (MoE) inference deployments previously had two scaling options: add a full model replica, or restart into a wider expert-parallel topology. SGLang's elastic expert-parallel scale-up adds GPU ranks to a running deployment, reserving metadata and communication buffers for max_ep_size at launch, admitting joiners through Mooncake PG, and carrying expert traffic over NIXL before committing the new topology. On 8×H100 with DeepSeek-V3-Lite FP8, scale-up reached the first completed request 5.3× to 8.2× faster than restart.

The SGLang team, with Google and Meta, released SGLang-TorchTPU, an open-source backend that runs the same SGLang serving engine on both GPUs and TPUs. TPU had not been a native PyTorch device, so a PyTorch-centric serving stack could not be reused directly; the new backend keeps the scheduler, RadixAttention and API server unchanged and places TPU work underneath. It is already running in production at Meta and serves as the rollout engine for reinforcement learning with TorchTitan through Miles.

The SGLang team, with Google and Meta, released SGLang-TorchTPU, an open-source backend that runs the same SGLang serving engine on both GPUs and TPUs. TPU had not been a native PyTorch device, so a PyTorch-centric serving stack could not be reused directly; the new backend keeps the scheduler, RadixAttention and API server unchanged and places TPU work underneath. It is already running in production at Meta and serves as the rollout engine for reinforcement learning with TorchTitan through Miles.

Meta released Muse Spark Open, an open-weight multimodal mixture-of-experts model, and SGLang and Miles added inference and reinforcement-learning training backends on launch day. Its text backbone has 816B parameters with about 42B active per token, accepting text, image, video and audio with a 1M-token context. On eight B300 GPUs with NVFP4 weights, an OpenHands multi-turn agentic coding replay reaches 80.3K total tokens/s per GPU at 61 tokens/s per user at concurrency 128.

Meta released Muse Spark Open, an open-weight multimodal mixture-of-experts model, and SGLang and Miles added inference and reinforcement-learning training backends on launch day. Its text backbone has 816B parameters with about 42B active per token, accepting text, image, video and audio with a 1M-token context. On eight B300 GPUs with NVFP4 weights, an OpenHands multi-turn agentic coding replay reaches 80.3K total tokens/s per GPU at 61 tokens/s per user at concurrency 128.