Driving VLMs stop distilling every token alike: pick the critical phrases, then trust the teacher only where it agrees
Synopsis
Knowledge distillation for driving vision-language models previously matched teacher distributions uniformly across answer tokens, diluting planning-relevant semantics and passing on confident teacher errors. RSKD first localizes task-critical phrases in the ground-truth answer, such as traffic participants, states and spatial relations, then uses a frozen sentence encoder to score how semantically consistent the teacher's prediction is with each phrase, so consistent positions take more teacher-distribution supervision and inconsistent ones take more ground-truth supervision. On DrivingVQA, LingoQA and MapLMv2, RSKD reaches 16.60, 0.672 and 81.82, beating the compared SFT, GKD and ToDi student baselines on the two QA benchmarks and nearly matching SFT's 81.85 on MapLMv2.
Interpretation
Distilling a multimodal driving model no longer treats every answer token the same; it first finds the phrases that actually matter for driving reasoning, then decides whether the teacher's wording at those positions is worth learning. Earlier distillation matched teacher distributions token by token, or adjusted supervision by teacher-student divergence and token difficulty, without judging whether those tokens carry task-critical driving semantics. RSKD reaches 16.60 on DrivingVQA and 0.672 on LingoQA, above SFT at 14.06 and 0.631, GKD at 16.34 and 0.640, and ToDi at 16.35 and 0.649; on MapLMv2 RSKD scores 81.82 against SFT's 81.85.
The teacher annotates critical phrases inside the ground-truth answer rather than generating a new answer, and a frozen sentence encoder scores the semantic similarity between teacher prediction and ground-truth phrase. That phrase-level consistency score is broadcast to the aligned tokens and moves the two supervision sources in opposite directions: consistent critical tokens get stronger JSD distillation, inconsistent ones get stronger ground-truth supervision. The student is Qwen3-VL-4B-Instruct and the teacher is an internally fine-tuned XEmbodied built on Qwen3-VL-30B-A3B-Instruct; the student is adapted with LoRA on language-model linear layers, all three benchmarks share one training mixture, and each configuration is repeated with random seeds and reported as a mean.
Ablations compare components under one reduced training budget, and removing routing in the ground-truth branch causes the largest drop. Replacing critical phrases with random answer spans lowers the score from 0.480 to 0.474, and removing consistency estimation gives 0.475, showing semantic importance alone is insufficient. All ablations run on the same 50% LingoQA subset and are compared only within that setting; disabling SFT routing drops the score to 0.468.
Perspective
The demonstrated scope is evaluation of a single teacher-student pair on the DrivingVQA, LingoQA and MapLMv2 driving QA benchmarks, with the training mixture and hyperparameters selected on validation splits; broader validation is left for future work.
Semantic consistency estimation depends on teacher phrase extraction and a frozen MiniLM encoder, so routing is sensitive to extraction and embedding errors; fixed similarity thresholds and rule-based contradiction detection may need recalibration on datasets with different response styles; evaluation currently covers only one teacher-student pair.