AI4S News · AI and science frontiers

Segmentation prompts stop being one vector: [SEG] attention maps lift Sa2VA on object parts and multiple objects

Language-guided segmentation usually compresses the referent's location into a single [SEG] hidden-state vector handed to the decoder, a vector with no spatial index while the decoder never sees the expression. AttnSeg instead passes the [SEG] token's attention over image patches as the decoder's dense prompt and supervises that map directly with an attention-guidance objective in two training stages. On Sa2VA, the 4B model raises ReasonSeg gIoU from 57.70 to 64.11 and GS-Eval from 61.91 to 72.11, with gains concentrated on object parts, multiple objects and single objects, and negligible change on stuff regions.

arXiv Sep 28, 2026

SWiM distills working memory so multimodal LLMs improve reasoning segmentation even without memory at inference, reaching state-of-the-art on benchmarks

The work finds that multimodal large language models benefit from using their own self-generated reasoning traces and localization proposals as working memory when revisiting the same image and query, and proposes SWiM: rollouts are selected by segmentation quality to build working memory, the memory-conditioned model serves as a teacher giving token-level distributional supervision along student-generated trajectories while the student receives only the original image and query, and on-policy self-distillation is jointly optimized with outcome-based reinforcement learning, achieving state-of-the-art performance on reasoning segmentation benchmarks.

Cilin Yan（First author）

Working Memory Distillation Reasoning Segmentation SWiM (Reasoning Segmenter with Working Memory)Multimodal Large Language Models On-Policy Self-Distillation