Muse Spark Open decoding passes 1,000 tokens per second: one resident kernel per layer lifts throughput from 517 to 1,109 tok/s
Synopsis
Muse Spark Open decoding rises from a 517 tok/s baseline to 1,109 tok/s by fusing each layer's attention and MoE into a single kernel that stays resident on the GPU, overlapping weight loads, tensor-core work and cross-GPU communication. Previously SGLang launched about 1,450 kernels per decoding step, and the GPU drained and refilled between them, leaving some SMs and threads idle. The figure was measured on Blackwell GPUs at batch size one with DFlash V2 speculative decoding and has not yet been extended to larger batches.
Interpretation
Muse Spark Open decoding throughput rises from a 517 tok/s baseline to 1,109 tok/s, crossing one thousand tokens per second. The baseline is the ordinary SGLang serving path, which even with speculative decoding launches about 1,450 kernels per decoding step; the GPU drains and refills between kernels, leaving some SMs and threads idle. Both numbers come from decoding measurements of the same model on Blackwell GPUs, 517 tok/s for the baseline and 1,109 tok/s for the megakernel, a before-and-after comparison under one setting.
The approach keeps one kernel resident on every SM and lets it carry several jobs of a layer at once instead of launching one kernel per operator. The megakernel is multiprogramming moved onto the GPU: each warp is given a fixed role at compile time, some fetching the next operation's weights early, some running tensor cores, some sending and receiving. Each launch spans one layer's attention through the next layer's qkv projection and KV write, with one twelve-warp thread block on each of 144 SMs, launched in clusters of four.
Weight prefetch and speculative expert prefetch spend otherwise idle waiting time moving data. Weights do not depend on the previous layer, so they can be read into shared memory or L2 while the current operation is still finishing; experts must wait for the router, so the slices of the 14 most-used experts on this GPU are pulled into L2 based on the previous step's routing. Expert prefetch saves about 0.2 ms out of 5.3 ms of step time, and a wrong guess costs only bandwidth because the GPU is idle anyway.
Perspective
The demonstrated scope is single-request decoding on Blackwell GPUs at batch size one with DFlash V2 speculative decoding, for Muse Spark Open split over eight GPUs with tensor parallelism; support for larger batches is explicitly listed as next work.
At one request no single overlap strategy works everywhere: whether a weight goes into shared memory, into L2 or both, how far ahead it goes and which cache hint it uses were each decided per operation by trying versions and keeping the fastest; the authors also note the kernel still resembles multiprogramming with a fixed number of tasks rather than run-time scheduling.