Omnimodal models mishear and missee without retraining: an inference-time nudge back to the relevant modality lifts CMM overall accuracy from 70.5% to 85.9%

When audio and video interfere, an omnimodal large language model can produce a wrong joint prediction even though the single-modality branch that the question actually needs still recovers the correct answer, a recoverable failure pattern the authors call modality misrouting. Earlier inference-time mitigation methods mainly suppress hallucinated outputs or recalibrate how much each modality contributes, and paid less attention to cases where the evidence is still present but does not reach the joint prediction. On the Visual Dominance and Audio Dominance subsets of CMM and the two cross-modal hallucination tasks of AVHBench, withholding the competing modality for each incorrect joint prediction recovers the ground-truth answer at rates of 52.9% to 74.4% on CMM and 37.3% to 46.2% on AVHBench.

CaRe acts only when the joint prediction and the candidate unimodal branch make a hard conflicting decision, then pushes the fused representation toward that branch while keeping both audio and video in the inference path. Instead of switching to the unimodal answer, it uses the discrepancy between the counterfactual and fused representations as a residual direction, scales the intervention by counterfactual confidence, and lets the strength decay with decoder depth. Across Qwen2.5-Omni-3B/7B and MiniCPM-o-2.6, CMM overall accuracy reaches 82.5%, 85.9% and 86.9%, and AVHBench overall accuracy reaches 81.0%, 82.3% and 80.2%, all above the unmodified backbones and the four inference-time baselines VCD, ICT, AVCD and MAD.

The method targets discrete, scorable answer spaces, so it fits audio-visual question answering where the task supplies candidate answers and question-relevant evidence remains retrievable from a modality-specific branch; on CMM and AVHBench it acts on the final three decoder layers of Qwen2.5-Omni-3B/7B and MiniCPM-o-2.6, modifying only the last prompt-token hidden state during prefilling and leaving generation untouched. The authors propose future work on softer conflict signals, more general routing criteria for open-ended generation, and more efficient single-pass implementations.

The authors note that CaRe may be less effective when relevant evidence is weakly encoded, when modality misrouting does not appear as an explicit prediction conflict, or when the task offers no readily scorable candidate answer space, and that the text-only routing probe and counterfactual branches add inference overhead. The routing signal shows model-specific sensitivity to prompt wording and option ordering, with 85 of 96 configurations reaching at least 95% agreement and the default configuration at 99.5% to 100%.