A 501B sparse model gets day-one serving support: coding agents tuned it themselves, reaching 708 tokens per second per request

A 501B-parameter mixture-of-experts model with 23B active parameters per token gets full day-one support in a serving engine, with reasoning, tool calling and structured outputs available and deployment across tensor, data or expert parallelism. The starting implementation was an almost unoptimized, nearly pure PyTorch version with no operator fusion for this architecture. On TP8 with GB200, seven changes brought batch-1 decode from 8.95 to 3.30 milliseconds per step and an 8×2048-token prefill from 328 to 176 milliseconds.

Coding agents carried out the optimization autonomously, profiling the model, writing kernels and benchmarking every change. Hand tuning usually has engineers locate hotspots one at a time, whereas here profiling, kernel writing and benchmarking run as an agent loop. The changes fuse post-norm, scaled residual and next pre-norm into one Triton kernel, fuse per-head Q/K normalization with RoPE or SSMax, and move both tensor-parallel all-reduces inside the normalization kernel.

The model ships with a DFlash2 draft head that predicts a whole block of tokens in one forward pass from the target model's hidden states instead of drafting token by token. Token-by-token drafting offers one candidate per step, while parallel drafting gives several at once but loses some left-to-right dependency. The drafter uses five sliding-window layers reading hidden states from five target layers, shares the target's embedding and LM head, and restores dependency with a grouped convolution and a small candidate selector; on the low-entropy SPEED-Bench category a single request reaches 708 tokens per second at acceptance length 3.7.

The report addresses inference engineers deploying this model, covering a single-node TP4/TP8 low-latency setup and a data-parallel attention with expert-parallelism high-throughput setup. The 1.71× prefill gain at high concurrency was measured on four GB300 GPUs with 32K input tokens and one output token. Adaptive verification pays off above concurrency 32 and is roughly neutral at low concurrency.

The drafter's high acceptance length comes from the low-entropy SPEED-Bench category, mostly code completion and text sorting, and lower acceptance is expected on open-ended chat. At concurrency 128 adaptive verification lowers acceptance length from 3.68 to 3.23, trading a little acceptance for throughput. Seven kernels in the optimized launch sequence still come from libraries, and most remaining time sits there.