Diffusion LLM guidance drops the extra forward pass: GCG edits attention-head features and lifts LLaDA-8B's Ruler score from 26.2 to 29.5
Synopsis
Diffusion language models generate text in parallel through iterative denoising, but existing guidance methods build a weak model and pay an extra forward pass at every denoising step, doubling inference cost. GCG first shows these models lean on local context and underuse distant evidence during denoising, then amplifies the outputs of attention heads that carry global context directly in feature space, without a weak model. On LLaDA-8B, Dream-7B and LLaDA-1.5 it beats U-CFG and A-CFG across several benchmarks, while LLaDA-8B keeps the same FLOPs as the unguided baseline.
Interpretation
Guiding a diffusion language model no longer requires running a second weak model: GCG edits attention-head features within a single forward pass and raises global-context use. Earlier U-CFG and A-CFG build a weak model by replacing prompt tokens or remasking low-confidence tokens, so each denoising step needs two forward passes, one for the base model and one for the weak model, doubling inference cost. On LLaDA-8B, GCG raises GSM8K-CoT from 46.7 to 48.4 and Ruler from 26.2 to 29.5 at 15.9T FLOPs, the same as the unguided baseline, whereas U-CFG and A-CFG cost 31.8T FLOPs.
The method first finds which attention heads pull in distant evidence, then subtracts the part of their output that overlaps local heads and amplifies the remainder. Global and local heads are ranked by the center of mass of their attention distributions, the attention-weighted average distance between query and key tokens, where a larger value means stronger attention to distant positions. Heads and target blocks are chosen once from 63 LongBench calibration samples and then fixed for all evaluation benchmarks; on LLaDA-8B, projecting global-head outputs onto the subspace spanned by local heads and amplifying the residual adds about 0.003% FLOPs.
The direction of guidance matters: reversing the same operation to strengthen local context, or applying it to blocks insensitive to global context, performs worse. Local context guidance (LCG) and least-sensitive blocks (LSB) are two control variants that reverse the guidance direction and swap the target blocks, respectively. On LLaDA-8B, LCG reaches 47.6 on GSM8K-CoT and 26.0 on Ruler, and LSB reaches 23.4 on Ruler, all below GCG's 48.4 and 29.5; Dream-7B and LLaDA-1.5 show the same trend.
Perspective
Evaluation covers three diffusion language models, LLaDA-8B, Dream-7B and LLaDA-1.5, on ARC, PIQA, HumanEval, MBPP, GSM8K-CoT, Ruler and MiniLongBench; global heads and target blocks must be selected once per model, which took about 2 to 3 hours on 8 NVIDIA A6000 GPUs in the experiments.
The authors note that selecting global heads, local heads and target blocks needs a separate calibration process of about 2 to 3 hours per model, and that improving it would raise practicality, especially for larger diffusion language models; they also state that GCG operates on existing models and may inherit their biases, privacy risks and hallucinations.