SWiM distills working memory so multimodal LLMs improve reasoning segmentation even without memory at inference, reaching state-of-the-art on benchmarks
The work finds that multimodal large language models benefit from using their own self-generated reasoning traces and localization proposals as working memory when revisiting the same image and query, and proposes SWiM: rollouts are selected by segmentation quality to build working memory, the memory-conditioned model serves as a teacher giving token-level distributional supervision along student-generated trajectories while the student receives only the original image and query, and on-policy self-distillation is jointly optimized with outcome-based reinforcement learning, achieving state-of-the-art performance on reasoning segmentation benchmarks.