Kimi K3 decode path written with coding agents: 842 tokens/s on 16 GB300 GPUs, 1.73x over stock vLLM

Kimi K3 decode no longer runs as a chain of separate operators but as one fused CuTe DSL path spanning attention, MoE and cross-GPU communication. Stock vLLM launches these stages separately, recomputes router scores and latent input projection on every GPU, and keeps attention output projection apart from its all-reduce. At batch 1 on 16 GB300 GPUs with DSpark, seven draft tokens and synthetic acceptance length 6, aggregate decode throughput is 842 tokens/s, 1.73x stock vLLM with the same speculative decoding setup.

Kernel shapes switch with batch size, and computation is overlapped with communication. Small and large matrix multiplications use GPU resources differently, and GPUs previously sat idle while communication completed. Attention input projections use small-matrix kernels at batches 1-2 and split-K kernels at larger batches; attention output projection fuses into its all-reduce, small batches merge producer and consumer into one kernel, and larger batches use a two-stage reduction to cut inter-GPU traffic.

The MoE stage splits router scoring and latent input projection across tensor-parallel ranks and combines each rank's share into one matrix multiplication. Every GPU previously computed full router scores and latent input projection, repeating computation, weight reads and kernel launches before experts ran. The shared expert runs on a separate stream overlapping routing or routed-expert work under a tuned schedule, and small-batch inference prefetches upcoming weights on a side stream to raise cache hit rates.

The optimization ships inside a validation process rather than as a speedup number alone. Line-by-line review of every kernel and every supported configuration is slow, while skipping it risks shipping buggy code to production. The process confines changes to model code, names supported hardware and workloads, and runs both kernel tests and model evaluations; an audit found the compiler moving activation reads ahead of a synchronization wait, fixed by restricting that load to static weights.

The demonstrated scope is TP8 and TP16 on NVIDIA GB300, decode batches up to 64, speculative decoding with various draft lengths, and decode context parallelism within one NVLink domain; throughput figures are aggregate decode throughput for the whole tensor-parallel group and exclude prefill. The 1.8x P90 interactivity on AgentX compares the optimized 16-GPU GB300 run with an earlier vLLM result on 8 GB300 GPUs. The isolation rule deliberately excludes optimizations to shared components such as input preparation, sampling and KV cache management.

The compiler reordering found in the audit shows simple functional checks can miss synchronization errors, and long-context and long-generation evaluations exposed attention and communication numerical issues that standalone kernel tests with fixed launch timing did not reproduce. The AgentX comparison uses different GPU counts on each side, with no same-hardware control reported. The fast-track process remains a proposal, and whether validation and maintenance effort stays bounded as contributions grow is still open.