Diffusion LLMs lose accuracy fast at high parallelism; RePara trains two-token blocks and lifts GSM8K from 28.20 to 68.54
Synopsis
Diffusion language models lose much of their accuracy when committing many tokens per step, and RePara reinforces short consecutive token blocks with group-relative policy optimization, raising LLaDA's GSM8K accuracy at Diff#16 from JustGRPO's 28.20 to 68.54.
Interpretation
A diffusion language model can predict several tokens in one forward pass, but treating that parallel prediction as a reliable multi-token commitment makes accuracy collapse as more tokens are committed per step on mathematics and code. Reasoning evaluations typically use only one or two tokens per diffusion step, and JustGRPO trains with single-token autoregressive-style rollouts while keeping parallel diffusion inference. Under a fixed 512-token generation budget, LLaDA, Dream, SDAR and LLaDA2.1-mini were evaluated at fixed token quotas, and at Diff#16 every model-benchmark combination scored below 10.
RePara has the model generate a short run of consecutive tokens in one forward pass during training, then optimizes the joint probability of that whole block using advantages derived from final-answer correctness. JustGRPO commits one token per rollout step, whereas RePara treats two consecutive tokens as a single action and puts the training objective directly on multi-token actions. Training ran for 128 updates with 256-token rollouts on LLaDA and SDAR, with the main comparison using fixed two-token blocks (Fix2) and evaluation on GSM8K and MATH-500.
The block structure also cuts the number of forward passes for both rollout generation and log-probability recomputation, because all tokens in a block share the same conditioning state. Single-token generation needs forward passes counted per token, while RePara counts them per block. In the LLaDA/MATH setting, Fix2 reduced core training-step time by 49.5% relative to JustGRPO, and a speedup remained after including checkpoint and metric-preparation costs.
Perspective
The work demonstrates parallel-decoding gains for mathematical reasoning on GSM8K and MATH-500 with two model families, LLaDA and SDAR, using 256-token training rollouts and downstream evaluation budgets no larger than 512 tokens. It is aimed at researchers and engineers working on post-training and inference acceleration for diffusion language models. Longer reasoning traces remain untested.
The mathematical gains hold on GSM8K and MATH-500, but direct application to code generation substantially reduces accuracy, and easier training data or initialization from a JustGRPO checkpoint yields only conditional gains. RePara mitigates the parallelism gap, yet LLaDA accuracy still declines when decoding increases from eight to sixteen tokens per step. Ground-truth-prefix probes characterize local prediction quality without establishing joint block correctness or isolating the causes of the coding regressions.