Eight AMD MI355X GPUs match NVIDIA B300 on Kimi K3: 24.4% more tokens per GPU, 87.4% more per dollar

An eight-GPU AMD Instinct MI355X system serving Kimi K3 on the AgentX agent workload exceeds NVIDIA HGX B300 in total tokens per second per GPU, and the gap widens when converted to tokens per dollar of modeled ownership cost. Published dense low-precision matrix peaks are 10.1 PFLOP/s MXFP4 for MI355X versus 13.5 PFLOP/s FP4 for B300, with both at 288 GB HBM3E and 8 TB/s peak bandwidth, so the serving gap is not set by paper throughput. At the 50 tokens/s/user P90 interactivity target, MI355X serves about 7,567 tokens/s/GPU against 6,083 for B300, 24.4% higher; at modeled $1.50 versus $2.26 per GPU-hour, that is 18.16 million versus 9.69 million tokens per dollar of TCO, 87.4% more.

Kimi K3 inference carries three kinds of state at once: a compressed KV history that grows with the conversation, fixed-dimension recurrent state, and token-routed expert networks. Its 24 MLA layers retain a compressed KV history that grows with the conversation and a new query still consults it, while its 69 KDA layers carry forward fixed-dimension state matrices updated as tokens arrive, so the two impose different cache and scheduling demands. Under TP8 each GPU handles 12 of Kimi K3's 96 MLA query heads but stores the entire compressed KV history; for a 142,000-token conversation the estimated MLA payload is 24 layers x 142,000 positions x 576 FP8 bytes, about 1.96 GB per GPU, which DCP8 cuts to about 245 MB by storing one eighth of the cached positions.

Cache reuse, MoE deduplication, kernel fusion and KDA parallelization together remove repeated work from each inference step. In the original ROCm path the routed-expert output projection ran on all eight GPUs, the single-token KDA path ran three compute kernels and two output copies, and the matrix-multiplication dispatcher packed inputs before choosing a kernel, leaving the copy unused when it selected the fallback. Dividing output coordinates across GPUs and reusing the existing all-reduce raised output throughput 3.05-4.91% in the PR's test on eight MI355X GPUs at TP8, concurrency 8-128 with random 8K-input/1K-output requests and prefix caching disabled; AttnRes fusion raised throughput 14.2% on 100K-input/1K-output requests; the fused KDA path performs the operations in one launch.

The result is specific to AgentX, a public agent workload that replays real coding-agent session traces while retaining request lengths, timing and prefix-sharing patterns, and runs with synthetic tokens and a fixed synthetic acceptance reference. The throughput and cost numbers come from eight-GPU, TP8 configurations with FP4 weights and FP8 KV, interpolated within each published offload or GPU-only series at the 30, 50 and 150 tokens/s/user interactivity targets, where concurrency, DCP and draft depth can vary between bracketing runs. The cost comparison uses SemiAnalysis's large-hyperscaler ownership model and published GPU cost constants, with some capital and utilization assumptions unpublished.

The public curves compare complete serving stacks, while the measurements beside each optimization come from separate tests and explain individual mechanisms under their stated configurations. Some changes landed after the image used for the headline result; their separate AgentX experiment found less than 1% higher throughput at concurrency 32 and 48 but 28.9% at 70, where the reported prefix-cache hit rate rose from 74.8% to 80.1%, and repeated-run uncertainty was not reported. Replay uses a downloaded dataset revision that is unrecorded, and closed-loop replay can produce different completed request mixes.