AI4S News · AI and science frontiers

PyTorch Nov 11, 2026

Meta's 816B sparse model now runs on one four-chip TPU host: a single whole-step kernel cuts per-token time 1.94× for one request

Meta's open-weights mixture-of-experts model Muse Spark Open has 816B total parameters and about 42B active per token; serving it on TPU v7 previously took 2.9× the time needed just to stream its weights from HBM for one verify step. SGLang-TorchTPU now writes all 62 layers of a decode or verify step as one Pallas program, issuing weight, KV and cross-device transfers ahead of the compute that needs them. Per output token, a single request drops from 3.78 ms to 1.95 ms, two concurrent requests gain 1.85×, and the gain narrows as concurrency grows.

Megakernel SGLang-TorchTPU Muse Spark Open Speculative Decoding KV Cache

SGLang moves its serving stack onto TPU: one engine for GPU and TPU, already in production at Meta

The SGLang team, with Google and Meta, released SGLang-TorchTPU, an open-source backend that runs the same SGLang serving engine on both GPUs and TPUs. TPU had not been a native PyTorch device, so a PyTorch-centric serving stack could not be reused directly; the new backend keeps the scheduler, RadixAttention and API server unchanged and places TPU work underneath. It is already running in production at Meta and serves as the rollout engine for reinforcement learning with TorchTitan through Miles.