Segmentation prompts stop being one vector: [SEG] attention maps lift Sa2VA on object parts and multiple objects

The interface of language-guided segmentation moves from a single [SEG] hidden-state vector to the [SEG] token's attention over the patch grid, so the decoder receives a prompt indexed by image location. The dominant prior interface compressed the referent's location into a vector with no spatial index, and the decoder, which never sees the expression, had to guess the referent's granularity on its own. On a frozen Sa2VA-4B, the same U-Net probe reading attention yields more gIoU than reading the hidden state on GS-Eval and ReasonSeg; adding the RGB image improves the hidden-state probe and leaves the attention probe unchanged.

A learned convolution with signed weights combines the per-head attention maps into one map that enters the decoder's dense prompt channel without thresholding or sampling. Prior methods that start from attention or similarity maps mostly sample sparse point prompts or threshold the map, discarding graded confidence. After training about a fraction of the weights are negative, and the positive mass concentrates in the last third of the layers, matching the late-layer concentration the probe measured.

Training runs in two stages: the VLM first learns to localize under the attention-guidance objective without the decoder, then the VLM and combination weights are frozen and only the decoder is trained to refine. The prior interface was supervised only through the mask loss computed after the decoder, so the decoder absorbed as much localization as it could. At 4B, against the base model finetuned on the same data for the same budget, AttnSeg still gains 4.19 and 9.56 gIoU on ReasonSeg and GS-Eval, and the attention-guidance objective accounts for most of the total gain.

The demonstrated scope is Sa2VA at the InternVL2.5-4B and InternVL3-14B scales, evaluated on the ReasonSeg test split and GS-Eval, trained on the thirteen-dataset mixture released with Sa2VA, with neither benchmark's training data in the mixture. The intended readers are researchers and engineers working on referring segmentation and multimodal grounding.

Gains are negligible on stuff regions and clear on object parts, multiple objects and single objects; the authors note that a stronger decoder (SAM3 replacing SAM2) is slightly above 4B AttnSeg on ReasonSeg but lower on GS-Eval. They also state that a benchmark gIoU improvement is not evidence the model resolves ambiguous or contested descriptions correctly, and that reading the attention requires recomputing the [SEG] rows, so results are not bitwise identical to a run using eager attention throughout.