# Attention Heads

3 related articles

Subscribe to updates

arXiv Nov 7, 2026
Diffusion LLM guidance drops the extra forward pass: GCG edits attention-head features and lifts LLaDA-8B's Ruler score from 26.2 to 29.5
Diffusion language models generate text in parallel through iterative denoising, but existing guidance methods build a weak model and pay an extra forward pass at every denoising step, doubling inference cost. GCG first shows these models lean on local context and underuse distant evidence during denoising, then amplifies the outputs of attention heads that carry global context directly in feature space, without a weak model. On LLaDA-8B, Dream-7B and LLaDA-1.5 it beats U-CFG and A-CFG across several benchmarks, while LLaDA-8B keeps the same FLOPs as the unguided baseline.
Diffusion language models generate text in parallel through iterative denoising, but existing guidance methods build a weak model and pay an extra forward pass at every denoising step, doubling inference cost. GCG first shows these models lean on local context and underuse distant evidence during denoising, then amplifies the outputs of attention heads that carry global context directly in feature space, without a weak model. On LLaDA-8B, Dream-7B and LLaDA-1.5 it beats U-CFG and A-CFG across several benchmarks, while LLaDA-8B keeps the same FLOPs as the unguided baseline.
Yonsei University
Jeimin Jeon（First author）
Global Context Guidance Diffusion Large Language Models Attention Heads
arXiv Oct 12, 2026
Same attention heads, new task, new role: one internal activation edit changes which fact the model reads and how it influences later layers
Across five open-weight models, holding graph facts fixed and changing only the task instruction redistributes how the same attention heads influence later computation; in Gemma-2-9B and Qwen3-14B, editing one early internal activation vector switches which fact the model reads while the written instruction and weights stay fixed.
University of Pennsylvania Yale University
Sovesh Mohapatra（First author）
Task-Vector Intervention Activation Patching Attention Heads Large language models
Transactions of the Association for Computational Linguistics Oct 7, 2026
A model splits a word's two meanings early in pretraining: two heads carry 14M, while 410M spreads the job across steadier ones
During pretraining of Pythia language models, word-sense disambiguation improves in sharp steps around 1,000 to 2,000 training steps, two layer-3 attention heads in 14M track that improvement and their ablation measurably hurts it, while 410M spreads the function across more heads that hold up better under position and part-of-speech perturbations.
University of California San Diego Rutgers, The State University of New Jersey
Pamela D. Rivière（First author）
Attention Heads Word Sense Disambiguation Pythia Language Models RAW-C Dataset Mechanistic Interpretability

You’re all caught up

Diffusion LLM guidance drops the extra forward pass: GCG edits attention-head features and lifts LLaDA-8B's Ruler score from 26.2 to 29.5

Same attention heads, new task, new role: one internal activation edit changes which fact the model reads and how it influences later layers

A model splits a word's two meanings early in pretraining: two heads carry 14M, while 410M spreads the job across steadier ones