Diffusion language models learn to count characters: no fine-tuning, summaries land within 3.1 characters of target
Synopsis
For summarization settings such as headlines and previews that must fit an exact character count, Delta estimates the expected character length of the current output inside a masked diffusion language model's denoising loop and dynamically inserts or removes masked positions to steer generation toward the target. It needs no fine-tuning and applies to off-the-shelf LLaDA and Dream. Across the XSum, CNN/DailyMail and arXiv summarization benchmarks, character-length error falls by an order of magnitude relative to the baselines while summary quality stays competitive under automatic and human evaluation.
Interpretation
Summaries can now be pushed close to a target character count at inference time without changing model weights or retraining. Earlier length control either relied on task-specific fine-tuning or on prompting and post-hoc filtering, which under autoregressive decoding achieved only approximate compliance. Across three summarization benchmarks, Delta brings mean absolute character-length error down to 2.3 to 3.1 characters, whereas the strongest autoregressive length-control baseline, LA-SF-AR, places only 84.2% of outputs within 10 characters of target on XSum and drops to 22.9% on arXiv.
The method continuously estimates how many characters will ultimately be written during denoising and adjusts the masked slots still to be filled accordingly. Masked diffusion models iteratively refine a whole buffer of masked tokens, so inserting or removing masks does not disturb an already written prefix the way autoregressive decoding would. Each position contributes either the exact character count of a committed token or a probability-weighted expected count over its current prediction, and the algorithm removes the least-confident masks or inserts new ones until the expected deficit is minimized, adjusting only right after a commitment step to prevent oscillation.
The gain in length precision comes from dynamic buffer adjustment itself rather than from differences in model or prompt. Turning off mask insertion and removal under the same model, prompt, confidence schedule and initialization falls back to fixed-buffer decoding. The fixed-buffer ablation reaches 92.31 to 113.94 characters of error on arXiv versus 2.8 to 3.1 for Delta, with broadly similar summary quality; in a blind evaluation of 50 CNN/DailyMail examples, Delta+Dream averages 4.27, against 4.32 for LA-SF-AR and 4.31 for GPT-4o-mini.
Perspective
The demonstrated scope is English summarization with two masked diffusion models in the 7–8B parameter range, LLaDA and Dream, with target lengths taken from reference summaries and evaluation on the XSum, CNN/DailyMail and arXiv benchmarks.
Inference speed is an explicit cost: on XSum Delta takes 0.67 to 0.69 seconds per summary, faster than LA-SF-AR's 1.43 seconds, but on CNN/DailyMail and arXiv it needs 8.59 seconds and 20.75 to 22.80 seconds, slower than LA-SF-AR's 2.19 and 13.22 seconds. Automatic quality scores are sensitive to the LLM judge, with GPT-4o-mini and GPT-5-mini differing substantially on consistency and coherence; the human evaluation covers only 50 CNN/DailyMail examples with two annotators. Thresholds were chosen by grid search per dataset and model, with no sensitivity analysis or adaptive setting method provided.