Diffusion-model RL stops scoring whole answers: part-of-speech credit lifts MATH500 by 9.0 points over the base model at the smallest budget
Synopsis
Diffusion language models commit tokens over many denoising steps, so a whole-response reward says little about which tokens helped later refinement. DTCA estimates each token's contribution from the confidence gain at still-masked positions after that token is committed, averages these signals by part-of-speech category into stable weights, and uses them to pick denoising timesteps and assign token-level learning signals, with entropy-based calibration. On LLaDA-8B-Instruct it beats the base model at generation budgets of 128, 256 and 512, by 9.0, 6.8 and 4.8 points on MATH500 and 5.3, 2.1 and 2.5 points on GSM8K.
Interpretation
Reinforcement learning for diffusion language models now places learning signals on individual tokens rather than only on the whole response. Earlier approaches mainly built learning signals from denoising trajectories or intermediate states, while a sequence-level reward alone does not identify which commitments help subsequent denoising. On LLaDA-8B-Instruct, DTCA improves over the base model by 9.0, 6.8 and 4.8 points on MATH500 and by 5.3, 2.1 and 2.5 points on GSM8K at generation budgets of 128, 256 and 512.
Contribution estimates are computed offline on denoising rollouts and merged by part-of-speech category into a fixed contribution table. Estimating token effects online requires extra forward passes and varies across rollout samples, so the estimates are made offline and averaged within each category. Offline estimation uses a denoising budget of half the maximum generation length, commits two tokens per step and evaluates each token with additional forward passes; the table is built from MATH and GSM8K training rollouts and stays fixed during training.
The contribution weights drive both timestep selection and token-credit scaling, and entropy calibration then adjusts these signals by the entropy change induced by policy updates. Timesteps are ranked by a score weighting token confidence with the part-of-speech contribution weight, and the smallest set covering 70% of the total score is selected, bounded between 2 and 32 steps. At the 256 budget, removing contribution credit drops MATH500 and GSM8K by 3.8 and 2.4 points, while removing entropy calibration drops them by 1.4 and 1.4 points; at the 512 budget, removing entropy calibration drops them by 4.2 and 3.5 points.
Perspective
The results are final-answer accuracy for LLaDA-8B-Instruct on MATH500 and GSM8K at generation budgets of 128, 256 and 512, with training on the MATH training split and a contribution table built from offline denoising rollouts over the MATH and GSM8K training splits.
Contribution estimation relies on the quality of part-of-speech alignment and annotation, and the contribution definition still lacks a more principled theoretical formulation; experiments focus on mathematical reasoning with LLaDA-8B-Instruct, and applying DTCA to tasks such as code generation or other diffusion language model architectures still needs evaluation.