Multimodal training rewrites text-only reasoning: removing one vision–language direction hurts spatial tasks but leaves logic intact
Synopsis
Whether multimodal training changes how a model reasons over text alone had been supported mainly by behavior. Across 10 multimodal configurations from 6 model families, this work builds a vision–language direction and removes each hidden state's projection onto it at inference; in Chameleon-7B, BBH navigate drops while boolean expressions stay largely intact, a double dissociation. The effect is direction-specific and layer-localized, and extends to mental-simulation tasks such as mental rotation, paper folding and perspective taking.
Interpretation
Multimodal training changes not only how a model handles images but which internal representation pathway it routes through when reading text alone. Earlier evidence that multimodal training affects text-only reasoning was behavioral or confined to synthetic mid-scale settings, and did not test whether visual-spatial structure is causally recruited without image input. Across 9 configurations where the vision–language axis is positive, removing that axis damages spatial-imagery tasks while symbol-manipulation controls are largely preserved; in Chameleon-7B, BBH navigate drops while boolean expressions barely move, a double dissociation.
The direction is not interchangeable with an arbitrary perturbation; it corresponds to a specific computational site inside the model. If the effect were generic damage from removing hidden-state energy, any direction of matched energy should produce a similar outcome. On the Chameleon-7B L0–7 block, energy-matched random directions yield a far smaller double-dissociation difference than the vision–language axis; the effect peaks within a few layers and tracks where each architecture feeds visual information into the language backbone.
In unified models with both understanding and generation pathways, the intervenable spatial carrier concentrates in the generation pathway. The understanding pathway also holds spatial information, but it was unclear whether that information forms a selectively intervenable causal handle. BAGEL's one-dimensional generation-path axis yields a spatial-versus-control double dissociation while the understanding-path axis is near zero; on a 27-task suite, Janus-Pro independently reproduces a large navigate drop with boolean expressions nearly unchanged.
Perspective
The conclusions rest on 10 multimodal configurations and linear or low-rank interventions, with tasks drawn from BBH, bAbI, SpartQA, StepGame, a mental-simulation battery and a controlled visual-spatial battery, making the analysis diagnostic rather than a complete taxonomy. Mental-simulation evidence is measurable only in Qwen3-VL configurations that clear a capability floor; Chameleon-7B sits at or below chance on those tasks.
The carrier is spatially concentrated rather than exclusively spatial: long-horizon non-spatial state tracking over 8 to 12 updates partially overlaps the same carrier. The analysis may miss nonlinear, highly distributed or cross-attention-specific mechanisms, and attribution for the Qwen family remains constrained by public uncertainty about pretraining data. The interventions test necessity, not whether the model builds an image-like scene, and do not locate the circuits that write and read the signal.