Visual latents were mostly decoration: AnchorSim makes perturbing them change answers, averaging 62.4 across nine benchmarks
Synopsis
Interleaved multimodal reasoning inserts continuous visual latent blocks between text spans, but direct perturbation shows that dropping, replacing or shuffling these blocks barely changes the model's own answer probability, while the same perturbations to the textual reasoning trace hurt far more. The authors call this mental simulation collapse and introduce AnchorSim, which during training contrasts predictions with the visual pathway against predictions with it masked, forcing the latents to carry answer-relevant information. Built on Qwen2.5-VL-7B, AnchorSim-7B averages 62.4 across nine visual reasoning benchmarks, 13.5 points above the base model, and visual perturbations become consequential again.
Interpretation
The visual latent blocks produced by interleaved multimodal reasoning models contribute far less to the model's own final answer than the textual reasoning segments between them. These architectures assume continuous visual latent blocks can serve as an internal visual workspace the model consults during reasoning, so visual information need not be compressed entirely into text. Perturbation scoring on Mirage-7B, SkiLa-7B, CoVT-7B and Monet-7B, each on its corresponding benchmark, leaves answer log-probability close to the unperturbed reference after visual blocks are dropped, replaced across samples or shuffled, whereas comparable perturbations to text reasoning blocks cause larger drops.
AnchorSim turns whether the visual pathway is used into a training objective, so generated visual latent blocks again affect predictions. Earlier two-stage training supervised only text likelihood, and visual alignment constrained latents to resemble helper-image embeddings without requiring the decoder to rely on them. For each step, next-token distributions are computed from the full context and from a context with all preceding visual latents masked, a Jensen–Shannon divergence penalizes agreement between them with stop-gradient on the masked branch, and reasoning text is distilled from a teacher that is an exponential moving average of the student and receives the ground-truth answer in its prompt.
Across nine public visual reasoning benchmarks, AnchorSim-7B reaches an average of 62.4, the best among the compared models. Its base Qwen2.5-VL-7B averages 48.9, and the interleaved Stage 1 model averages 59.7. AnchorSim-7B ranks first on every benchmark except BLINK and V*, where it ranks second; in closed-loop perturbation, perturbing visual latents flips a stated fraction of answers and lowers accuracy, while the interleaved-only baseline barely flips under visual perturbation.
Perspective
The demonstrated scope is 7B-scale Qwen2.5-VL-class multimodal models on nine public visual reasoning benchmarks, together with perturbation re-checks of the visual and textual pathways in those settings; the authors propose testing scalability to larger models and real-world multimodal decision-making.
The authors state that AnchorSim is evaluated mainly at the 7B scale on benchmark-style tasks, leaving scalability to larger models and real-world decision-making untested; latent content is defined operationally as answer-relevant, image-conditioned information, without requiring individual latent tokens to correspond to image crops or be directly decodable.