NEO-VLA drops the vision encoder: action signals peak before the final visual layer
Synopsis
NEO-VLA removes the fixed visual encoder from vision-language-action models: images are tokenized by lightweight patch convolutional layers and evolve with language through the backbone, and a flow-matching action expert draws on instruction-conditioned visual representations at different depths. In controlled comparisons the native architecture beats encoder-based counterparts; full fine-tuning narrows but does not erase the gap, and extra scale alone cannot recover it. Layer-wise probing shows action-critical signals often peak before the final visual representation.
Interpretation
Vision-language-action models no longer compress perception into a fixed visual endpoint before action reasoning begins; NEO-VLA lets image representations evolve with language through the backbone, and the action expert draws on them at different depths. The prior setup compressed perception through a visual encoder first, and action reasoning started only after its output. In controlled comparisons the native architecture outperforms encoder-based counterparts; full fine-tuning narrows the gap but does not erase it, and additional scale alone cannot recover it.
Images are first tokenized by lightweight patch convolutional layers, then pass through the backbone together with language, and a flow-matching action expert reads instruction-conditioned visual representations at different depths. The flow-matching action expert here draws on layered visual representations that evolve with computation, not on a single encoder output. Layer-wise probing shows action-critical signals often peak well before the final visual representation.
Across diverse manipulation settings and distribution shifts, NEO-VLA delivers stronger and more robust control. This contrasts with the default premise that action policies should inherit a fixed visual endpoint. The results span diverse manipulation settings and distribution shifts, supporting the direction of drawing visual representations across the computational hierarchy.
Perspective
This report rests on summary-level material covering controlled comparisons, a full fine-tuning control, layer-wise probing, and control performance across diverse manipulation settings and distribution shifts; specific tasks, datasets and metric values are not given in the material.
The material states that full fine-tuning narrows but does not erase the gap between native and encoder-based architectures, and that additional scale cannot recover it; layer-wise probing shows action-critical signals often peak before the final visual representation, but the material does not say whether that peak location is stable across tasks.