Driving-video motion questions stump six video-language models under prompting, but light finetuning nearly solves them
Synopsis
On CARLA third-person driving videos, six open-source video-language models judging forward/backward, left/right, and accelerate/decelerate stayed near chance under standard, visual, few-shot, and chain-of-thought prompting, with systematic class biases such as every model scoring 100% on forward and 0% on backward. LoRA finetuning then reached near-perfect held-out accuracy for all models except VideoLLaMA2 on accelerate/decelerate, and simple classifiers on pre- and post-compression visual features decoded motion cues at high accuracy. The motion information is present inside the models, but the prompted prediction pathway does not reliably use it.
Interpretation
Basic ego-motion judgments in driving video are unreliable for off-the-shelf video-language models, and prompt-level techniques do not consistently fix them. Video QA benchmarks had reported steady gains, which invites reading fluent outputs as temporal reasoning; this work narrows the question to three binary motion pairs: forward/backward, left/right, and accelerate/decelerate. Standard prompting leaves most models near 50%, and the bias is systematic: all evaluated models score 100% on forward and 0% on backward, collapsing to the forward class; InternVL 3.5 reaches 67% on left/right overall while its two classes score 39% and 95%.
Motion information has not vanished inside the models, since light supervision recovers it. Prompting failure does not mean the architecture lacks the representation; the bottleneck lies in supervision and access. With 400 videos per motion pair and an 80/20 held-out split, LoRA finetuning reaches near-perfect accuracy except for VideoLLaMA2 on accelerate/decelerate, and one-layer CNN probes on frozen pre- and post-compression features far exceed prompted inference, for example lifting Qwen2.5-VL and Qwen3-VL forward/backward from about 50% to 97.5%.
Where motion information goes in the visual pipeline is architecture-dependent, but the common finding is a gap between internal decodability and final prompted prediction. It was previously unclear whether failure occurs in visual encoding, visual-to-language compression, or language generation. VideoLLaMA2 often drops after its spatio-temporal connector, Qwen2.5-VL and Qwen3-VL improve markedly after visual merging, InternVL 3.5 is already highly decodable before merging, and MiniCPM-V 4.5 rises from 67.5% to 100% on forward/backward.
Perspective
The evaluation covers six open-source video-language models: VideoLLaMA2, Qwen2-VL, Qwen2.5-VL, Qwen3-VL, InternVL 3.5, and MiniCPM-V 4.5. Simulated data are 1,200 CARLA third-person videos at 60 FPS, 200 per motion class; real-world data are 60 in-car first-person videos at 25 FPS, 10 per class split between day and night, used only for standard prompting. LoRA and feature probes use an 80/20 train/held-out split over 400 videos per motion pair, leaving 80 held-out clips per pair. The work suits researchers and engineers working on driving perception, video-language model evaluation, and multimodal reasoning.
The real-world evaluation is small, so its reported accuracies are inherently less stable; the authors treat it mainly as evidence that the same biases appear outside CARLA and note that the class-level correlation is inflated by the forward/backward collapse and should not be read as an independent sim-to-real generalization measure. Few-shot results use 199 query clips per class, giving a binomial standard error of roughly 3.5 percentage points near chance, so differences of a few points should not be over-interpreted. The authors also state that without single-frame or frame-shuffling controls, high LoRA and probe accuracy cannot by itself establish that the cues are genuinely temporal, since some performance may reflect per-frame appearance cues in the CARLA renders, which is left to future work.