Continuous diffusion language models drop rare words: GRAFT lifts ELF-B's rare-word share from 0.42 to 0.82 of the human rate without retraining
Synopsis
Continuous diffusion language models generate a whole sentence in parallel without drawing a word at intermediate steps, and the cost is a collapsed lexical tail: ELF nearly matches human text on common words, yet its rarest words appear at only a fraction of the human rate, a gap that persists from 105M to 652M parameters and under every sampler setting tested. GRAFT reads the decoding head mid-generation, commits the rare words the average is about to discard, and regenerates from noise, raising the rare-word share from 0.42–0.77 to 0.82–1.25 of the human rate across three ELF scales and the independent LangFlow implementation, at the cost of a small rise in non-word rate on some models.
Interpretation
Continuous diffusion language models match human text on common words but visibly lack rare ones, and the gap survives larger models and sampler tuning. A same-size GPT-2 moves its rare-word share from zero to above the human rate by its decoding rule alone, whereas ELF's full sweep over steps, guidance, temperature and scale never reaches the human rate. Against same-distribution human text from ELF's training corpus, ELF-B's rare-word share is 5.01% versus 11.88% for human text, and the gap does not close from 105M to 652M parameters.
The loss happens inside the denoising head, which regresses to an average embedding over competing candidates, so a rare word is averaged away when a stronger common competitor is present. The decoding head itself does not favor common words: fed clean human-text embeddings, it recovers nearly all of the original words. On ELF-B the denoising head errs more often than the decoding head on the rarest words while matching it on common words; among rare positions, loss is far lower in the lowest decile of strongest-competitor probability than in the highest.
GRAFT changes no weights: it reads the decoding head once mid-generation, commits rare words that still hold substantial probability but are losing to common competitors, and regenerates from the same noise. Continuous diffusion models previously had no interface that turns a distribution into a discrete word choice before the final step, so temperature scaling, sampling and beam search could not recover the lost tail. On ELF-B the rare-word share rises from 5.01% to 9.76% (0.82 of the human rate) while the non-word rate falls from 2.64% to 2.57%; on LangFlow the tuned dose reaches 12.23%, or 1.03 of the human rate.
The gain comes from which rare word is committed, not from restarting or from extra compute. An argmax control sharing GRAFT's positions, mechanics, dose and compute is null on every column, moving ELF-B's rare-word share only from 5.01% to 5.31%. At equal budget, best-of-4 stock sampling reaches only 5.97% on ELF-B against GRAFT's 9.76%, and on LangFlow stock sampling catches up only at best-of-16.
Perspective
The demonstrated setting is unconditional generation with frozen ELF-B/M/L (105M, 342M and 652M parameters) and LangFlow (652M), against a human reference from the same OpenWebText shard; translation is largely unaffected because rare content can be copied from the source, and summarization is essentially unchanged at the default selection threshold.
The authors report that no system closes the gap to the human non-word rate (2.57%–3.79% versus 1.14%), that LangFlow's tuned dose reaches 1.03 of the human rare-word rate while its non-word rate rises to 3.79%, that ELF-L's JSD improvement falls within seed-to-seed variation, and that MAUVE falls monotonically with dose; no large-scale human preference study was run.