# Discrete Diffusion

7 related articles

Subscribe to updates

arXiv Nov 10, 2026
Discrete diffusion drops offline training: DMBD optimizes discrete variables directly and beats sampling baselines in 14 of 15 multi-robot task-allocation settings
Discrete diffusion for combinatorial optimization has relied on an offline-trained neural denoiser, while training-free model-based diffusion works only on continuous variables and must round its solutions. DMBD replaces the denoising step with Monte Carlo sampling weighted by the known objective to approximate categorical reverse transitions directly; on a heterogeneous multi-robot coalition-formation and scheduling benchmark it reaches the lowest makespan among sampling-based methods in 14 of 15 settings and beats all baselines in 8.
Discrete diffusion for combinatorial optimization has relied on an offline-trained neural denoiser, while training-free model-based diffusion works only on continuous variables and must round its solutions. DMBD replaces the denoising step with Monte Carlo sampling weighted by the known objective to approximate categorical reverse transitions directly; on a heterogeneous multi-robot coalition-formation and scheduling benchmark it reaches the lowest makespan among sampling-based methods in 14 of 15 settings and beats all baselines in 8.
Haejoon Lee（First author）
Discrete Model-Based Diffusion Combinatorial Optimization Multi-Robot Task Allocation Sampling-Based Optimization Discrete Diffusion
arXiv Oct 14, 2026
Phylogenetic posteriors get an iterative generative model: it matches IQ-Tree on simple data and beats it on complex evolution
Posterior inference over phylogenetic trees has depended on likelihoods, which become intractable once real processes such as insertions, deletions or site selection are modeled. This family of conditional generative models builds a tree step by step, bottom-up, middle-out or top-down, and is trained with a discrete diffusion objective; on real and synthetic data it recovers MCMC posteriors more closely than the prior simulation-based method Phyloformer 2, matches IQ-Tree on simple data, and is reliably more accurate on more realistic complex models.
Posterior inference over phylogenetic trees has depended on likelihoods, which become intractable once real processes such as insertions, deletions or site selection are modeled. This family of conditional generative models builds a tree step by step, bottom-up, middle-out or top-down, and is trained with a discrete diffusion objective; on real and synthetic data it recovers MCMC posteriors more closely than the prior simulation-based method Phyloformer 2, matches IQ-Tree on simple data, and is reliably more accurate on more realistic complex models.
New York University
Ethan Baron（First author）
Simulation-Based Inference Discrete Diffusion Phylogenetic Inference
arXiv Oct 14, 2026
Discrete diffusion need not pay for vocabulary size: masked and uniform noising both hit minimax rates on sparse data
Discrete diffusion models have lacked the statistical theory that continuous diffusion enjoys, and it was unclear whether they can exploit sparse structure in the data distribution. This analysis casts masked-diffusion score estimation as a per-context label-smoothed classifier and shows the smoothing is equivalent to regularizing with the score loss of the uniform distribution. On sparse distributions where at most k sequences have positive probability, the terminal laws of both masked and uniform noising attain the sparse minimax rate, and the smoothing level can be chosen adaptively from the number of distinct training sequences without knowing the sparsity parameter.
Discrete diffusion models have lacked the statistical theory that continuous diffusion enjoys, and it was unclear whether they can exploit sparse structure in the data distribution. This analysis casts masked-diffusion score estimation as a per-context label-smoothed classifier and shows the smoothing is equivalent to regularizing with the score loss of the uniform distribution. On sparse distributions where at most k sequences have positive probability, the terminal laws of both masked and uniform noising attain the sparse minimax rate, and the smoothing level can be chosen adaptively from the number of distinct training sequences without knowing the sparsity parameter.
Université Paris Sciences et Lettres Centre National de la Recherche Scientifique Institut national de recherche en sciences et technologies du numérique
LUCA BLUNDA（First author）
Discrete Diffusion Minimax Optimal Estimation Rates Label Smoothing Regularization Density Estimation
arXiv Oct 30, 2026
Discrete diffusion sampling stops rerolling at every jump: exponential residual flows make the trajectory deterministic after initialization and enable consistency distillation
Sampling from discrete diffusion models previously redrew randomness at every step, so the trajectory could not be fixed. Exponential residual flows give each position and token a continuous residual variable that evolves deterministically by piecewise ODEs after initialization, with a token change triggered whenever a residual reaches zero, and the construction is proved to preserve the marginals of the original diffusion at every time. At matched model-evaluation budgets, generation quality is comparable to ancestral sampling, and the deterministic trajectories support consistency distillation.
Sampling from discrete diffusion models previously redrew randomness at every step, so the trajectory could not be fixed. Exponential residual flows give each position and token a continuous residual variable that evolves deterministically by piecewise ODEs after initialization, with a token change triggered whenever a residual reaches zero, and the construction is proved to preserve the marginals of the original diffusion at every time. At matched model-evaluation budgets, generation quality is comparable to ancestral sampling, and the deterministic trajectories support consistency distillation.
École Pour l'Informatique et les Techniques Avancées Bocconi University
Samson Gourevitch（First author）
Exponential Residual Flow Discrete Diffusion Consistency Distillation
arXiv Oct 30, 2026
Diffusion language models without pretraining from scratch: LoRA-only conversion turns four autoregressive models diffusive in hours on one GPU
BYOD is a LoRA-only recipe for converting autoregressive language models into diffusion language models: every base parameter stays frozen, each layer's causal mask becomes bidirectional, and a masked-diffusion objective trains at most 102.4M tokens in 3.1–4.0 hours on one Google Colab GPU, turning Gemma 2 9B, Llama 3.1 8B, Qwen2.5 7B and Ministral 8B into models that generate from fully masked sequences, with adapters at 4.1–5.8% of final parameters. The strongest conversion, BYOD-Gemma, reaches 50.6% average accuracy on nine tasks against 51.9% for LLaDA 8B Instruct in the same harness; open-generation quality at low NFE budgets remains the main limitation.
BYOD is a LoRA-only recipe for converting autoregressive language models into diffusion language models: every base parameter stays frozen, each layer's causal mask becomes bidirectional, and a masked-diffusion objective trains at most 102.4M tokens in 3.1–4.0 hours on one Google Colab GPU, turning Gemma 2 9B, Llama 3.1 8B, Qwen2.5 7B and Ministral 8B into models that generate from fully masked sequences, with adapters at 4.1–5.8% of final parameters. The strongest conversion, BYOD-Gemma, reaches 50.6% average accuracy on nine tasks against 51.9% for LLaDA 8B Instruct in the same harness; open-generation quality at low NFE budgets remains the main limitation.
Diffusion Language Model BYOD: Build Your Own DLM Low-Rank Adaptation Parallel Decoding Discrete Diffusion
arXiv Oct 2, 2026
Discrete diffusion for categorical MRFs: a pinning decomposition and weight-sharing score learner yield end-to-end sample complexity bounds
For categorical distributions with local dependence modeled by low-order Markov random fields, the work introduces a pinning decomposition of the discrete score, builds a weight-sharing neural score learner combined with tau-leaping into an end-to-end sampling procedure, derives optimal sampling guarantees from finite data with explicit dependence on vocabulary size, MRF interaction order, and sample size, and reports numerical experiments on Potts, Ising, and tree-structured models where weight-sharing score networks outperform fully connected ones for sampling long sequences.
For categorical distributions with local dependence modeled by low-order Markov random fields, the work introduces a pinning decomposition of the discrete score, builds a weight-sharing neural score learner combined with tau-leaping into an end-to-end sampling procedure, derives optimal sampling guarantees from finite data with explicit dependence on vocabulary size, MRF interaction order, and sample size, and reports numerical experiments on Potts, Ising, and tree-structured models where weight-sharing score networks outperform fully connected ones for sampling long sequences.
University of Chicago
Shivam Kumar（First author）
Discrete Diffusion Sample Complexity Bounds Pinning Decomposition Weight-Sharing Neural Score Learner Markov Random Fields
arXiv Oct 2, 2026
E-MoE uses expert routing as a discrete latent, cutting LM1B one-step generative perplexity from 1433.8 to 643.8
The work proposes E-MoE, which treats the expert-routing decisions of a Mixture-of-Experts backbone as a discrete shared latent so that the masked diffusion reverse process becomes a mixture of factorized distributions, improving few-step generation without increasing active parameters per token: it beats factorized baselines on synthetic 2-D densities, binarized MNIST and LM1B, lowering LM1B generative perplexity at NFE=1 from MDLM's 1433.8 to 643.8.
The work proposes E-MoE, which treats the expert-routing decisions of a Mixture-of-Experts backbone as a discrete shared latent so that the masked diffusion reverse process becomes a mixture of factorized distributions, improving few-step generation without increasing active parameters per token: it beats factorized baselines on synthetic 2-D densities, binarized MNIST and LM1B, lowering LM1B generative perplexity at NFE=1 from MDLM's 1433.8 to 643.8.
Arseny Ivanov（First author）
E-MoE Mixture of Experts Masked Diffusion Models Discrete Diffusion Text Generation

You’re all caught up

Discrete diffusion drops offline training: DMBD optimizes discrete variables directly and beats sampling baselines in 14 of 15 multi-robot task-allocation settings

Phylogenetic posteriors get an iterative generative model: it matches IQ-Tree on simple data and beats it on complex evolution

Discrete diffusion need not pay for vocabulary size: masked and uniform noising both hit minimax rates on sparse data

Discrete diffusion sampling stops rerolling at every jump: exponential residual flows make the trajectory deterministic after initialization and enable consistency distillation

Diffusion language models without pretraining from scratch: LoRA-only conversion turns four autoregressive models diffusive in hours on one GPU

Discrete diffusion for categorical MRFs: a pinning decomposition and weight-sharing score learner yield end-to-end sample complexity bounds

E-MoE uses expert routing as a discrete latent, cutting LM1B one-step generative perplexity from 1433.8 to 643.8