Katsuya Ogata

Ogata and Nakashima ask where encoding happens when multimodal language models skip dedicated perceptual encoders and expose a shared transformer to lightly projected patches, audio frames, or discrete visual tokens, and through linear probing, similarities to perceptual encoders, and causal analyses they find the transformer internalizes the missing computation, constructing task-usable perceptual representations within its own early-to-middle layers, a structure they call a Virtual Encoder.

Ogata and Nakashima ask where encoding happens when multimodal language models skip dedicated perceptual encoders and expose a shared transformer to lightly projected patches, audio frames, or discrete visual tokens, and through linear probing, similarities to perceptual encoders, and causal analyses they find the transformer internalizes the missing computation, constructing task-usable perceptual representations within its own early-to-middle layers, a structure they call a Virtual Encoder.