Meta's 816B sparse model now runs on one four-chip TPU host: a single whole-step kernel cuts per-token time 1.94× for one request

On a single four-chip TPU v7 host, one decode or verify step of Muse Spark Open now runs as one kernel covering all 62 layers, so weight movement from HBM to the matrix units is no longer interrupted at kernel boundaries. The earlier native path split each layer into several kernels plus collectives, and every boundary stopped the weight stream, so a verify step took 2.9× the time needed to stream its weights. On GSM8K with DFlash2 speculative decoding on both paths, one request takes 1.95 ms against 3.78 ms per output token, two concurrent requests gain 1.85×, and the gap narrows as concurrency rises.

The kernel schedules each step's data movement itself: dense weights stream through a tile ring prefetched across layer boundaries, expert weights are requested as soon as routing ends, and tensor-parallel partial sums are written straight into the owning device's memory by remote DMAs issued inside the kernel. The native path started a separate collective after each matmul, so weights and partial sums could not move until the previous step finished. About ten tiles in flight sustain 3.34 TB/s per device while a single DMA stream reaches only 1.13 TB/s, and a small remote round costs about 3 µs.

The attention loop drops the running max of online softmax because Muse Spark applies a weight-free RMS norm to q and k and gives q a fixed gain, bounding every attention score to magnitude 43.8, far from f32 exponential overflow. The running max chained each KV chunk to the previous one, since a chunk could not start its exponentials until the prior chunk's maximum was known, costing about 0.17 µs per chunk in a microbenchmark. In a microbenchmark at production shapes, longer chunks with independent streams cut attention time 2.4×, and removing the running max took it to 3.9×; in the server, per-request decode speed rose 18% for two requests at 500K context.

The demonstrated setting is one four-chip TPU v7 host with 8 devices under tensor parallelism, the Muse Spark Open model, and the 1M-context trace set of AgentX; the GB200 comparison points come from InferenceX's published results as of 2026-11-02, with costs of $1.21 per TPU v7 chip-hour and $1.86 per GB200 GPU-hour.

Collective folding and sequence-parallel prefill change the order of some reductions, so results are not bit-identical, and the text says accuracy checks cover those changes; the pipelined KV writes and the split expert loop stay bit-identical. The whole-step kernel's gain over the native path narrows as concurrency grows, and prefill becomes the limit at high concurrency.