# AIR-MoE

The work introduces AIR-MoE, an inverted-index-inspired two-stage MoE routing architecture based on vector quantization: a first stage assigns tokens to VQ codewords to build a candidate expert set, and a second stage computes exact routing scores restricted to that shortlist, approximating true top-K routing while avoiding full expert scoring and imposing no structural constraints on expert parameters; the authors also provide a lower bound on the mass recall achieved by AIR-MoE, and empirically improve the perplexity-FLOPs trade-off over existing MoE routers in regimes with up to more than a million tiny experts while achieving consistent perplexity improvements over the best existing baseline.

The work introduces AIR-MoE, an inverted-index-inspired two-stage MoE routing architecture based on vector quantization: a first stage assigns tokens to VQ codewords to build a candidate expert set, and a second stage computes exact routing scores restricted to that shortlist, approximating true top-K routing while avoiding full expert scoring and imposing no structural constraints on expert parameters; the authors also provide a lower bound on the mass recall achieved by AIR-MoE, and empirically improve the perplexity-FLOPs trade-off over existing MoE routers in regimes with up to more than a million tiny experts while achieving consistent perplexity improvements over the best existing baseline.