# vLLM TorchTPU

Inferact and Google released vLLM TorchTPU, a backend that serves model definitions already in vLLM on Google Cloud TPUs without changing model code. Previously, bringing a model to TPU meant writing separate model code; this backend keeps vLLM's command line, OpenAI-compatible server, scheduler and KV cache manager, with TPU-specific work sitting underneath them. It compiles through torch.compile's TPU backend to XLA and uses custom JAX/Pallas kernels for attention, mixture-of-experts and quantized matmul.

Inferact and Google released vLLM TorchTPU, a backend that serves model definitions already in vLLM on Google Cloud TPUs without changing model code. Previously, bringing a model to TPU meant writing separate model code; this backend keeps vLLM's command line, OpenAI-compatible server, scheduler and KV cache manager, with TPU-specific work sitting underneath them. It compiles through torch.compile's TPU backend to XLA and uses custom JAX/Pallas kernels for attention, mixture-of-experts and quantized matmul.