# Pallas Kernels

Inferact and Google released vLLM TorchTPU, a backend that serves model definitions already in vLLM on Google Cloud TPUs without changing model code. Previously, bringing a model to TPU meant writing separate model code; this backend keeps vLLM's command line, OpenAI-compatible server, scheduler and KV cache manager, with TPU-specific work sitting underneath them. It compiles through torch.compile's TPU backend to XLA and uses custom JAX/Pallas kernels for attention, mixture-of-experts and quantized matmul.

Inferact and Google released vLLM TorchTPU, a backend that serves model definitions already in vLLM on Google Cloud TPUs without changing model code. Previously, bringing a model to TPU meant writing separate model code; this backend keeps vLLM's command line, OpenAI-compatible server, scheduler and KV cache manager, with TPU-specific work sitting underneath them. It compiles through torch.compile's TPU backend to XLA and uses custom JAX/Pallas kernels for attention, mixture-of-experts and quantized matmul.

The SGLang team, with Google and Meta, released SGLang-TorchTPU, an open-source backend that runs the same SGLang serving engine on both GPUs and TPUs. TPU had not been a native PyTorch device, so a PyTorch-centric serving stack could not be reused directly; the new backend keeps the scheduler, RadixAttention and API server unchanged and places TPU work underneath. It is already running in production at Meta and serves as the rollout engine for reinforcement learning with TorchTitan through Miles.

The SGLang team, with Google and Meta, released SGLang-TorchTPU, an open-source backend that runs the same SGLang serving engine on both GPUs and TPUs. TPU had not been a native PyTorch device, so a PyTorch-centric serving stack could not be reused directly; the new backend keeps the scheduler, RadixAttention and API server unchanged and places TPU work underneath. It is already running in production at Meta and serves as the rollout engine for reinforcement learning with TorchTitan through Miles.