# Masked Diffusion Models

Post-training for masked diffusion language models relies on Monte Carlo estimates of a blockwise surrogate objective, and the existing scheme samples the block index at random, so one block may be evaluated repeatedly while others are never evaluated. A full-block estimator evaluates every block exactly once and averages the results; analysis shows it stays unbiased and removes the between-group variance from block selection. Across five benchmarks in mathematics, planning and coding it reaches the highest accuracy on four, averaging 12.79 points above the random block sampling baseline.

Post-training for masked diffusion language models relies on Monte Carlo estimates of a blockwise surrogate objective, and the existing scheme samples the block index at random, so one block may be evaluated repeatedly while others are never evaluated. A full-block estimator evaluates every block exactly once and averages the results; analysis shows it stays unbiased and removes the between-group variance from block selection. Across five benchmarks in mathematics, planning and coding it reaches the highest accuracy on four, averaging 12.79 points above the random block sampling baseline.

Transformer-based masked diffusion models now have an end-to-end statistical guarantee: when the data distribution is well approximated by a mixture of hidden Markov trees, the training samples needed for a target KL accuracy are governed by latent structural complexity rather than growing exponentially with the sequence space. The analysis covers both the parallelization error of parallel unmasking and the finite-sample estimation error of the mask predictor, and the transformer architecture and sampling procedure need no knowledge of tree topologies or component parameters, adapting to unknown latent structure.

Transformer-based masked diffusion models now have an end-to-end statistical guarantee: when the data distribution is well approximated by a mixture of hidden Markov trees, the training samples needed for a target KL accuracy are governed by latent structural complexity rather than growing exponentially with the sequence space. The analysis covers both the parallelization error of parallel unmasking and the finite-sample estimation error of the mask predictor, and the transformer architecture and sampling procedure need no knowledge of tree topologies or component parameters, adapting to unknown latent structure.

The work proposes E-MoE, which treats the expert-routing decisions of a Mixture-of-Experts backbone as a discrete shared latent so that the masked diffusion reverse process becomes a mixture of factorized distributions, improving few-step generation without increasing active parameters per token: it beats factorized baselines on synthetic 2-D densities, binarized MNIST and LM1B, lowering LM1B generative perplexity at NFE=1 from MDLM's 1433.8 to 643.8.

The work proposes E-MoE, which treats the expert-routing decisions of a Mixture-of-Experts backbone as a discrete shared latent so that the masked diffusion reverse process becomes a mixture of factorized distributions, improving few-step generation without increasing active parameters per token: it beats factorized baselines on synthetic 2-D densities, binarized MNIST and LM1B, lowering LM1B generative perplexity at NFE=1 from MDLM's 1433.8 to 643.8.

The work factorizes masked diffusion language model (MDM) inference into five axes—score, cardinality, region, commitment, and planning—defines adaptation opportunity as the one-step utility advantage of the best candidate action over a validation-selected fixed action, finds across three models (LLaDA-8B-Instruct, LLaDA-1.5, Dream-7B) and ten tasks that adaptation opportunities are highly heterogeneous, and proposes selective adaptation in which lightweight detectors calibrated on validation prompts identify high-opportunity states, capturing for example 56.9% of the candidate-set oracle opportunity by adapting only the top 10% of states on LLaDA-8B constrained JSON filling, with end-to-end selective decoding raising task-macro utility from 0.392 to 0.424 (LLaDA-8B), 0.344 to 0.

The work factorizes masked diffusion language model (MDM) inference into five axes—score, cardinality, region, commitment, and planning—defines adaptation opportunity as the one-step utility advantage of the best candidate action over a validation-selected fixed action, finds across three models (LLaDA-8B-Instruct, LLaDA-1.5, Dream-7B) and ten tasks that adaptation opportunities are highly heterogeneous, and proposes selective adaptation in which lightweight detectors calibrated on validation prompts identify high-opportunity states, capturing for example 56.9% of the candidate-set oracle opportunity by adapting only the top 10% of states on LLaDA-8B constrained JSON filling, with end-to-end selective decoding raising task-macro utility from 0.392 to 0.424 (LLaDA-8B), 0.344 to 0.