Max Planck Society

RNA inverse folding has long relied on iterative optimization scored by an external folding algorithm, which is sensitive to initialization and prone to local optima. This work defines multinomial diffusion directly over nucleotide sequences, guides denoising with a target secondary structure, and compares a 2D contact-map projection with a purely 1D pairing-aware architecture. On the in-distribution test set, the 2D and 1D models reach median normalized RNAforester similarities of 0.82 and 0.79, while generated sequences stay close to random-sequence diversity.

RNA inverse folding has long relied on iterative optimization scored by an external folding algorithm, which is sensitive to initialization and prone to local optima. This work defines multinomial diffusion directly over nucleotide sequences, guides denoising with a target secondary structure, and compares a 2D contact-map projection with a purely 1D pairing-aware architecture. On the in-distribution test set, the 2D and 1D models reach median normalized RNAforester similarities of 0.82 and 0.79, while generated sequences stay close to random-sequence diversity.

The work introduces AIR-MoE, an inverted-index-inspired two-stage MoE routing architecture based on vector quantization: a first stage assigns tokens to VQ codewords to build a candidate expert set, and a second stage computes exact routing scores restricted to that shortlist, approximating true top-K routing while avoiding full expert scoring and imposing no structural constraints on expert parameters; the authors also provide a lower bound on the mass recall achieved by AIR-MoE, and empirically improve the perplexity-FLOPs trade-off over existing MoE routers in regimes with up to more than a million tiny experts while achieving consistent perplexity improvements over the best existing baseline.

The work introduces AIR-MoE, an inverted-index-inspired two-stage MoE routing architecture based on vector quantization: a first stage assigns tokens to VQ codewords to build a candidate expert set, and a second stage computes exact routing scores restricted to that shortlist, approximating true top-K routing while avoiding full expert scoring and imposing no structural constraints on expert parameters; the authors also provide a lower bound on the mass recall achieved by AIR-MoE, and empirically improve the perplexity-FLOPs trade-off over existing MoE routers in regimes with up to more than a million tiny experts while achieving consistent perplexity improvements over the best existing baseline.