# Masked Diffusion Language Model

2 related articles

Subscribe to updates

arXiv Oct 27, 2026
Masked diffusion models lock answers too early: HiLo-RD distills large-window trajectories, lifting GSM8K accuracy by 9.4 points
Masked diffusion language models can fix answer fragments before the supporting reasoning is resolved when the response window is small, and those fragments then become context for later predictions. HiLo-RD uses the same model's large-window trajectory as a frozen teacher and aligns its hidden representations to cropped small-window views at committed positions. At the same small inference budget, GSM8K accuracy rises from 68.84% to 78.24% while mean readable length falls from 362.8 to 271.3 tokens, and the gains extend to code generation without code-domain training.
Masked diffusion language models can fix answer fragments before the supporting reasoning is resolved when the response window is small, and those fragments then become context for later predictions. HiLo-RD uses the same model's large-window trajectory as a frozen teacher and aligns its hidden representations to cropped small-window views at committed positions. At the same small inference budget, GSM8K accuracy rises from 68.84% to 78.24% while mean readable length falls from 362.8 to 271.3 tokens, and the gains extend to code generation without code-domain training.
Zhejiang University Alibaba Group (China)Hong Kong University of Science and Technology Nanyang Technological University The Hong Kong University of Science and Technology (Guangzhou)
Xiang Li（First author）
Representation Distillation Masked Diffusion Language Model Self-Distillation Mathematical Reasoning Budget-Induced Premature Commitment
arXiv Oct 5, 2026
Pivot-SD supervises only high-impact commitments in denoising, letting LLaDA-8B-Instruct beat full-sequence SFT and budget-matched diffusion RL baselines on math and code with 200 questions
The work introduces Pivot-SD, an offline self-distillation framework that selects high-impact commitments (pivots) during denoising via an information-gain metric measuring uncertainty reduction over remaining masked positions, trains pivots from successful trajectories with cross-entropy and pivots from failed trajectories with targeted unlikelihood while leaving the rest of the failed trajectory untouched, and with only 200 questions and four rollouts each improves LLaDA-8B-Instruct over full-sequence SFT and budget-matched diffusion RL baselines across math and code benchmarks.
The work introduces Pivot-SD, an offline self-distillation framework that selects high-impact commitments (pivots) during denoising via an information-gain metric measuring uncertainty reduction over remaining masked positions, trains pivots from successful trajectories with cross-entropy and pivots from failed trajectories with targeted unlikelihood while leaving the rest of the failed trajectory untouched, and with only 200 questions and four rollouts each improves LLaDA-8B-Instruct over full-sequence SFT and budget-matched diffusion RL baselines across math and code benchmarks.
University of Toronto Korea Advanced Institute of Science and Technology
Seo Hyun Kim（First author）
Self-Distillation Pivot-SD Masked Diffusion Language Model

You’re all caught up

Masked diffusion models lock answers too early: HiLo-RD distills large-window trajectories, lifting GSM8K accuracy by 9.4 points

Pivot-SD supervises only high-impact commitments in denoising, letting LLaDA-8B-Instruct beat full-sequence SFT and budget-matched diffusion RL baselines on math and code with 200 questions