vLLM trains a DSpark drafter for Kimi K3: math-reasoning interactivity rises from about 110 to about 435 tok/s/user

Open-source speculative-decoding drafters can now cover frontier models at the 2.8-trillion-parameter scale: the Kimi K3 DSpark drafter produced through the extended Speculators library raises single-stream interactivity from about 110 to about 435 tok/s/user on math-reasoning workloads. Speculators had already validated training, packaging and deployment on Qwen3.6-35B-A3B, Gemma-4-31B-it, GLM-5.2 and others, but a single-node vLLM-plus-training setup hits VRAM limits at Kimi K3's scale even with 4-bit quantized weights. Under concurrent load, output throughput reaches up to about 3.5x at matched interactivity; raising concurrency from 1 to 16 lifts aggregate output throughput from 177 to 683 tokens per second while median time to first token grows only from 379 to 479 milliseconds.

The drafter keeps DFlash's single non-causal backbone pass that predicts a whole block of tokens, then adds two lightweight components to recover dependencies missing between parallel positions. DFlash's parallel positions cannot condition on one another, so they may favor conflicting continuations at the same position and produce mismatches; verification stops at the first rejected token, so one mistake also invalidates the remaining suffix, which the DSpark paper calls suffix decay. A Markov logit-bias head adjusts each position's logits using the previously selected token through a low-rank transition matrix without another transformer pass, while a confidence head estimates each token's acceptance probability and a hardware-aware scheduler uses those estimates to verify longer prefixes under light load and trim unlikely suffixes when busy.

For multi-node training, hidden-state extraction moves from a single node to a disaggregated architecture in which training and inference scale independently. The single-node scheme devotes half the GPUs to training and half to serving the target model in vLLM with on-demand hidden-state extraction, but Kimi K3 needs at least two GB300 nodes of four GPUs each to serve even at 4-bit quantization. The MooncakeHiddenStatesConnector uses the Mooncake transfer engine as its backend, with a master proxy process managing vLLM and training instances registered as clients and moving hidden states over RDMA or TCP; the best configuration found was three-node sets with two nodes for inference and one for training.

The demonstrated scope is the Kimi K3 DSpark drafter's inference behavior across nine evaluation domains including math reasoning, on LongBench-v2 long-context prompts, and at concurrency from 1 to 16, with training done on a GB300 NVL72 rack provided by Verda.

DSpark's semi-autoregressive design improves inference only when the extra sequential work stays much cheaper than another draft-model forward pass; the reported speedups and acceptance lengths come from the releasing party's own evaluation and deployment configuration.