A 501B sparse model moves into a production API: SGLang serves Beam natively at 127.6K tokens per second per GPU
Synopsis
Beam, Reflection's 501B-parameter mixture-of-experts model with 23B active parameters per token, targets software engineering and long-horizon agentic tasks and is planned for Apache 2.0 release. SGLang now supports Beam natively and powers its production API, turning inference efficiency into serving performance through BF16, NVFP4 and dynamic FP8 checkpoints, DFlash2 speculative decoding and prefix caching. On an OpenHands multi-turn agentic coding replay with the NVFP4 checkpoint on four NVIDIA GB300 GPUs, SGLang reaches 127.6K total tokens/sec per GPU at 185.87 tokens/sec per user (p50) with concurrency 32.
Interpretation
Hybrid-attention models like Beam now have a native serving path, so developers can deploy them across the runtimes and coding harnesses they already use. SGLang did not previously support Beam, and its production API lacked a matching serving stack. SGLang supports Beam natively and powers Reflection's production API, with BF16, ModelOpt NVFP4 and compressed-tensors dynamic FP8 checkpoints.
The serving stack must manage two KV lifetimes, because Beam's full-attention layers keep the complete history while sliding-window layers keep only the recent window. A model with a single attention type needs only one KV cache layout. Beam uses grouped-query attention with eight KV heads shared among 48 query heads, a 512-token sliding window and a 1,048,576-token context limit.
Throughput gains come from kernel-level fusion and scheduling, not from the sparse structure alone. Without fusion, normalization and residual paths repeatedly write and read intermediate results. Projection packing, fusion across attention and residual boundaries, a specialized BF16x3 GEMM for the FP32 router and GB300 MoE tuning together support the reported performance.
On realistic multi-turn coding load, cache reuse spreads the cost of long prefixes. Each conversation would otherwise reprocess an initial prompt of roughly 74K tokens every turn. The OpenHands replay uses 13 turns per conversation with about 220 output tokens per turn, reusing the prefix for a roughly 92% aggregate cache hit rate.
Perspective
The report covers Beam's integration, optimization and performance measurement on SGLang, for inference engineers deploying long-context agentic models. Performance numbers come from an OpenHands multi-turn coding replay with the NVFP4 checkpoint on four NVIDIA GB300 GPUs at TP4, sweeping concurrency 1, 4, 8, 16 and 32. A separate AgentX replay reports 83,504 tokens/sec/GPU at 43.9 tokens/sec/user at concurrency 96, and 348.0 tokens/sec/user for a single session.
The OpenHands measurements use eight-token DFlash2 blocks with acceptance length fixed at four by simulation alongside real draft tokens, so they do not measure natural draft acceptance or model accuracy. The NVFP4 expert-parallel MegaMoE path requires both the flashinfer_megamoe runner and the all-to-all backend, and BF16 and dynamic FP8 remain outside that extension. TPU support and SGLang-TorchTPU recipes are not yet available.