Speculative decoding pays off more than predicted on sparse MoE models: fine-grained models peak near batch size 1
Synopsis
Prior cost models estimated speculative decoding from the expert-weight bytes activated during verification, predicting a concave speedup curve with limited gains at low batch size for sparse MoE targets. Across 10 sparse MoE models from 30B to 1T parameters and 8 to 512 experts, fine-grained models instead often peak near batch size 1. Two effects explain it: routing overlap across adjacent draft positions cuts the unique experts loaded during verification by 21% to 34% relative to an independence baseline, and GPU execution dynamics, namely bandwidth-utilization gaps between MoE and attention kernels plus L2 reuse, decouple runtime from expert bytes.
Interpretation
Speculative decoding speedup on sparse MoE targets is not uniformly concave in batch size; fine-grained models with small experts stay efficient at low batch size and often peak near batch size 1. Earlier MoE speculative-decoding cost models estimated overhead from the expert-weight bytes activated during verification, and from that predicted speedup rising then falling with batch size, with limited gains at low batch size. Sweeping batch size over 10 sparse MoE models (30B to 1T parameters, 8 to 512 experts), sparse models such as Qwen3-235B and Kimi-K2.5 show non-concave curves, while Mixtral-8x7B and Qwen2-57B with larger experts remain concave; Qwen3-30B and Qwen3-235B keep similar curve shapes across several datasets.
When several draft positions are verified, adjacent positions select heavily overlapping experts, so a verification pass loads far fewer unique experts than independent routing would imply. The independence baseline uses only each expert's empirical selection probability and assumes positions do not influence each other, which overestimates the unique expert count during verification. Across eight measured models, the overlap beyond the independence baseline accounts for 69% to 92% of total overlap in 6 of them; verifying four tokens for Qwen3-30B loads only 2.4 unique experts per layer versus 3.6 predicted under uniform random routing, a 33% reduction.
At batch size 1, expert-weight bytes alone do not determine verification latency; GPU execution dynamics also contribute. The activated-expert-bytes model assumes runtime scales linearly with the expert-weight bytes loaded and ignores kernel efficiency and cache behavior. On H100, Qwen3-30B (FP8) MoE kernels reach 45% to 49% of peak HBM bandwidth while attention kernels reach only 22% to 25%, so MoE layers take 33% of forward-pass time despite holding 67% of weights; in the same setting the MoE L2 hit rate rises from 5.72% in autoregressive decoding to 27.79% under speculative verification.
A maximum-entropy Iterative Proportional Fitting (IPF) framework predicts unique expert counts from pairwise routing-overlap statistics and generates synthetic routing that reproduces speedup curves without training a model. Evaluating speculative-decoding target and draft architecture choices directly requires training and benchmarking, which is expensive at frontier scale. Skewed IPF predicts unique expert counts within 4% for every measured model (maximum 3.7% for Qwen3-Next-80B) and uniform IPF within 5%; replacing learned routing with synthetic routing under fixed execution conditions substantially narrows the gap to real-model curves and captures the qualitative low-batch trend.
Perspective
The conclusions rest on 10 sparse MoE models, the vLLM v1 inference stack and MT-Bench as the base dataset, measured mainly on H100, with Kimi-K2.5 and DeepSeek-V3 on B200; long context was checked only for Qwen3-30B with roughly 32k-token prompts, and the MoE draft-layer analysis is a synthetic design study rather than a measurement of trained drafters.
IPF matches only pairwise overlap statistics and does not model higher-order routing structure, position dependence in expert marginals, or how temporal overlap itself varies with architectural choices; residual gaps remain between synthetic routing and real-model curves. Which architectural factors determine temporal overlap, and how reliably overlap transfers across model families and sparsity patterns, remain open questions.