Azure Maia 300 targets useful tokens per watt: 96 accelerators per rack, 36% more system compute
Synopsis
Microsoft released Azure Maia 300, a third-generation in-house inference accelerator designed as a whole system, with software, silicon, rack, networking and datacenter co-designed so each watt and each dollar yields more useful tokens. Compared with the previous generation, it replaces uniform on-chip memory with a locality-aware non-uniform organization across chiplets, letting software place data near the compute that uses it, and adds narrow-precision formats with hardware data-type conversion to cut bytes moved per token. At system level the rack doubles to 96 accelerators, the scale-up domain grows to a 12,288-accelerator superpod, the multi-chiplet strategy raises system compute FLOPs by 36% and HBM bandwidth increases 2.5 times.
Interpretation
Azure Maia 300 targets cloud-scale agentic inference and treats the whole system, not a single chip, as the design object, co-designing software, silicon, rack, networking and datacenter to raise useful tokens per watt and per dollar. The earlier Maia 200 introduced Software Defined Locally Accessed Dataflow Architecture, or SDLA, separating control-path programming from data-path execution; Maia 300 evolves that architecture for long-context, reasoning-intensive and decode-heavy inference. Microsoft's stated comparison is that the multi-chiplet approach places compute on TSMC N2P, raising system compute FLOPs by 36%, adding 2.5 times HBM bandwidth and 14% more scale-up bandwidth.
Data placement moves from uniform on-chip memory to a locality-aware non-uniform organization across chiplets, so software can place data near the compute that uses it and move less of it. The previous SoC used a uniform memory access organization, leaving data location opaque to software; the new organization makes locality programmable. Supporting changes include new narrow-precision formats and hardware-accelerated data-type conversion, reducing bytes moved per token and allowing larger models, longer contexts and more concurrent sessions within the same power envelope.
The rack doubles in size and the scale-up domain expands, so more accelerators can work effectively as a single system. Previously the rack and scale-up domain were smaller, making it harder to use many accelerators as one system. The rack doubles to 96 accelerators and the scale-up domain grows to a superpod of 12,288 accelerators.
Perspective
This report draws on a summary of the Microsoft Azure blog and covers Maia 300's architectural direction, memory organization, precision formats and rack scale; it does not include specific benchmarks, latency figures or comparisons with other accelerators. It is aimed at researchers and engineers working on inference infrastructure and accelerator architecture.
The material reports architectural improvement ratios and scale figures, but does not state latency or throughput under real agentic workloads, nor comparisons with other inference platforms.