Arseny Ivanov

The work proposes E-MoE, which treats the expert-routing decisions of a Mixture-of-Experts backbone as a discrete shared latent so that the masked diffusion reverse process becomes a mixture of factorized distributions, improving few-step generation without increasing active parameters per token: it beats factorized baselines on synthetic 2-D densities, binarized MNIST and LM1B, lowering LM1B generative perplexity at NFE=1 from MDLM's 1433.8 to 643.8.

The work proposes E-MoE, which treats the expert-routing decisions of a Mixture-of-Experts backbone as a discrete shared latent so that the masked diffusion reverse process becomes a mixture of factorized distributions, improving few-step generation without increasing active parameters per token: it beats factorized baselines on synthetic 2-D densities, binarized MNIST and LM1B, lowering LM1B generative perplexity at NFE=1 from MDLM's 1433.8 to 643.8.