Ask a driving video which way the car moves, and six video-language models all say forward; light finetuning gets near-perfect
Lead
On binary ego-motion questions over CARLA driving videos, six video-language models under standard prompting collapse onto one class, answering forward for every forward/backward clip, while lightweight LoRA finetuning brings most of them near perfect, indicating the motion cues sit inside the models but the prompted pathway does not retrieve them.
Story
Asked whether the ego vehicle is going forward or backward, left or right, accelerating or decelerating, six open-source video-language models under standard prompting barely vary their answer with the clip and instead fall onto a fixed class. These models are increasingly used as general-purpose video reasoners, and benchmark scores keep rising, yet it was unclear whether they actually use temporal dynamics in safety-critical settings. On 1200 third-person CARLA driving videos, every evaluated model scores 100% on forward and 0% on backward, collapsing entirely to the forward class; left/right and accelerate/decelerate show model-dependent biases, for instance InternVL 3.5 reaching 67% on left/right from class accuracies of 39% and 95%.
Boxing a static reference object, giving demonstrations, or adding step-by-step reasoning does not reliably fix the bias, while lightweight finetuning or a small classifier on intermediate visual features brings the same models near perfect. Prompt-level interventions mainly make outputs more fluent and confident, and a few newer models improve markedly on selected action pairs, but the gain does not transfer across motion categories. LoRA finetuning, with 400 videos per action pair and an 80/20 split, reaches near-perfect accuracy except for VideoLLaMA2 on accelerate/decelerate; one-layer CNNs on frozen intermediate features lift Qwen2.5-VL and Qwen3-VL forward/backward from 50.00% and 55.00% to 97.50%, and left/right reaches 100% after merging.
What to watch
The evaluation covers one controlled family of tasks: binary forward/backward, left/right and accelerate/decelerate judgments on third-person CARLA driving video, plus 60 first-person real-world driving videos used only for standard prompting. The intended readers are researchers and engineers considering video-language models inside driving reasoning or interaction pipelines. The source proposes testing next whether these behaviors persist on larger real-world driving datasets such as nuScenes and the Waymo Open Dataset, and exploring architectures or training objectives that more directly preserve and use temporal information.
The source notes that without single-frame or frame-shuffling controls, the high LoRA and probe accuracies cannot by themselves establish that the cues are genuinely temporal, since some performance may reflect per-frame appearance cues in the CARLA renders, and this is left to future work. The real-world set has only 60 videos, 10 per class, so its accuracies are inherently less stable and too few for the supervised LoRA and feature-probing experiments. The source also states that the class-level sim-to-real correlation is inflated by the forward/backward collapse across models and should not be read as an independent measure of generalization.