Shiva Sreeram

On CARLA third-person driving videos, six open-source video-language models judging forward/backward, left/right, and accelerate/decelerate stayed near chance under standard, visual, few-shot, and chain-of-thought prompting, with systematic class biases such as every model scoring 100% on forward and 0% on backward. LoRA finetuning then reached near-perfect held-out accuracy for all models except VideoLLaMA2 on accelerate/decelerate, and simple classifiers on pre- and post-compression visual features decoded motion cues at high accuracy. The motion information is present inside the models, but the prompted prediction pathway does not reliably use it.

On CARLA third-person driving videos, six open-source video-language models judging forward/backward, left/right, and accelerate/decelerate stayed near chance under standard, visual, few-shot, and chain-of-thought prompting, with systematic class biases such as every model scoring 100% on forward and 0% on backward. LoRA finetuning then reached near-perfect held-out accuracy for all models except VideoLLaMA2 on accelerate/decelerate, and simple classifiers on pre- and post-compression visual features decoded motion cues at high accuracy. The motion information is present inside the models, but the prompted prediction pathway does not reliably use it.