After DeepSeek open-sourced DeepInfer, SGLang matches or beats it on its own benchmarks, up to 1.04x prefill throughput
Synopsis
DeepSeek open-sourced DeepInfer, the inference engine behind its own service, along with its kernels; SGLang then brought the Mega kernel series, DSpark speculative decoding, SWA bounded replay and both Engram placements into its mainline and compared against DeepInfer on the same hardware using DeepInfer's own benchmarks. On 4x GB300 and 4x B300, SGLang reaches up to 1.04x DeepInfer's prefill throughput, within 1% at the remaining points; on 64x GB300 its decode step is 1.1% to 7.4% shorter from 16 to 256 requests per GPU and on par at 8. SGLang also ships the KV cache system, prefill/decode disaggregation and cross-replica load balancing that DeepInfer's release notes list as not included.
Interpretation
SGLang can now run the inference optimizations DeepSeek built for DeepInfer, including the Mega kernel series, DSpark speculative decoding, SWA bounded replay and both Engram table placements. DeepInfer is the inference engine behind DeepSeek's own service, and its release notes list a KV cache system, prefill/decode disaggregation and cross-replica load balancing as components not included. The comparison uses DeepInfer's own prefill and decode benchmarks on the same hardware and weights; SGLang reaches up to 1.04x DeepInfer's prefill throughput on 4x GB300 and 4x B300, and shortens decode steps by 1.1% to 7.4% on 64x GB300.
The speed comes from cutting intermediate memory traffic and kernel launches rather than swapping in a different model. Mega-mHC feeds its output directly into MegaMoE, attention produces MXFP8 output for the output projection, and the prefill compressor and request-window KV copies use fused kernels. Breakable CUDA graphs cover prefill steps of up to 8K tokens per GPU, and decoder replay uses separate graphs sized to the bounded suffix so those layers still replay from CUDA graphs while the rest of a large prefill step runs eagerly.
SGLang provides, as a full serving system, the three components DeepInfer's release does not include. A unified radix cache reuses KV across requests and conversation turns, Linker connects L1 GPU caches to shared L3 storage such as 3FS or Mooncake, and HiCache also supports a host-memory L2 tier. Prefill and decode run on separate servers with Mooncake, NIXL or MORI as the KV transfer backend; a model gateway distributes requests across replicas and supports routing by cached prefix, serving on GB300, B300, B200, H200 and MI355X.
Perspective
The comparison covers DeepSeek-V4.1-Flash prefill on 4x GB300 and 4x B300 and decode on 64x GB300, using DeepInfer's own benchmarks and the same weights; settings shown, such as balanced expert routing, exist for this comparison only, and real deployment follows the cookbook page.
The two engines are measured differently: DeepInfer times prefill for about 9 batches on GPU 0, while SGLang averages every complete batch on every GPU between the points where 20% and 90% of requests finish; in the decode comparison SGLang keeps Engram tables on device while DeepInfer keeps them in host memory, and DeepInfer's draft model carries 64 redundant experts while SGLang's has none.