# Text Generation

Autoregressive language models normally learn next-token prediction in a fixed left-to-right order; Max(MI(S;t)) instead ranks target tokens by their mutual information with the source and already-selected targets before training, exposing informative targets earlier. On GPT-style models it reaches 94.96% average accuracy across arithmetic tasks, against 74.84% for next-token prediction and 86.29% for reverse order, close to the 97.33% upper bound from exhaustively testing 24 orders; on WikiText-2 it lowers average perplexity from 13.58 to 10.77. Only the training-target organization changes; architecture and autoregressive decoding stay the same.

Autoregressive language models normally learn next-token prediction in a fixed left-to-right order; Max(MI(S;t)) instead ranks target tokens by their mutual information with the source and already-selected targets before training, exposing informative targets earlier. On GPT-style models it reaches 94.96% average accuracy across arithmetic tasks, against 74.84% for next-token prediction and 86.29% for reverse order, close to the 97.33% upper bound from exhaustively testing 24 orders; on WikiText-2 it lowers average perplexity from 13.58 to 10.77. Only the training-target organization changes; architecture and autoregressive decoding stay the same.

The work proposes E-MoE, which treats the expert-routing decisions of a Mixture-of-Experts backbone as a discrete shared latent so that the masked diffusion reverse process becomes a mixture of factorized distributions, improving few-step generation without increasing active parameters per token: it beats factorized baselines on synthetic 2-D densities, binarized MNIST and LM1B, lowering LM1B generative perplexity at NFE=1 from MDLM's 1433.8 to 643.8.

The work proposes E-MoE, which treats the expert-routing decisions of a Mixture-of-Experts backbone as a discrete shared latent so that the masked diffusion reverse process becomes a mixture of factorized distributions, improving few-step generation without increasing active parameters per token: it beats factorized baselines on synthetic 2-D densities, binarized MNIST and LM1B, lowering LM1B generative perplexity at NFE=1 from MDLM's 1433.8 to 643.8.