Text-to-image attention stops hugging the first token: diffusion models pile it on modality boundaries and image corners

Attention sinks in text-to-image Transformers are not the fixed first-token pattern known from language models; their preferred positions follow the generation paradigm and the token layout. Attention sinks had been studied mainly in large language models, where causal masking makes early tokens persistent attention anchors. Position-wise sink ratios across FLUX.1, Z-Image and Infinity-2B put sequence position 0 at 37.52% for Infinity-2B, against 0.32% for FLUX.1 and 0.47% for Z-Image.

Sink positions become increasingly image-directed as generation proceeds, staying locally continuous between adjacent steps but drifting across stages. There was no unified comparison of whether sink positions move with the inference stage in text-to-image models. The image-directed ratio rises from 41.21% to 50.47% for FLUX.1, 7.46% to 12.73% for Z-Image and 81.24% to 89.95% for Infinity-2B; adjacent-step top-1 overlap is 79.12% and 76.03% for FLUX.1 and Z-Image, falling to 61.29% and 54.11% across full stages.

Sink-related heads carry real functional weight, while the effect of sink positions themselves differs by model. Earlier work lacked a controlled comparison that perturbs sink-related heads and sink positions separately. On Z-Image, pruning the top 50 sink-concentration heads drops the average metric by 2.36%, versus 1.84% for random 50 heads and 1.60% for the bottom 50; on Infinity-2B, removing the position-0 token drives GenEval from 64.56 to 11.37, while FLUX.1 keeps GenEval between 64.81 and 65.45 after removing candidate sink positions.

A single learnable register token can move part of the sink attention onto a non-semantic position without hurting generation quality. Register tokens had been used mainly in vision Transformers to absorb excess attention, not tested on text-to-image diffusion models. Adding one register token to the FLUX.1 image-token sequence raises GenEval from 67.14 to 68.85 and DPG from 83.67 to 84.01, while the two recurring native sink positions weaken from 12.95% to 12.44% and from 3.04% to 2.72%.

The study covers three text-to-image Transformers, FLUX.1, Z-Image and Infinity-2B, evaluated on the GenEval and DPG-Bench prompt benchmarks under official inference settings, with all experiments run on NVIDIA H100 GPUs. The analysis targets inference-time attention behavior; the authors leave broader architectures, scales and training recipes for future validation, and name understanding how attention sinks emerge during training as the next step.

The authors note that several frequent sink positions also coincide with modality boundaries, corner positions or global-scale tokens, so score changes are read as pathway sensitivity rather than isolated causal effects of sink behavior alone. Prompt distribution mainly affects where stable sink heads route attention, not which heads become sink-dominant; because the two datasets differ in prompt length, semantics and complexity, the authors do not attribute the change to a single prompt factor. The register token produces partial redistribution of sink dominance, not full replacement of the native sinks.