Through path-intervention and representation analyses, this work identifies a question-relay mechanism in audio-visual large language models, where question-position hidden states carry both required-modality evidence and interfering-modality cues, and proposes SECRET, a training-free method that steers question states toward required-source evidence using source-conditioned positive and negative question representations, achieving higher accuracy than prior training-free methods on CMM and AVHBench across three AVLLMs, with gains of up to 18.0 and 7.1 percentage points over base models, and reducing distractor overlap in modality-specific captioning.
Through path-intervention and representation analyses, this work identifies a question-relay mechanism in audio-visual large language models, where question-position hidden states carry both required-modality evidence and interfering-modality cues, and proposes SECRET, a training-free method that steers question states toward required-source evidence using source-conditioned positive and negative question representations, achieving higher accuracy than prior training-free methods on CMM and AVHBench across three AVLLMs, with gains of up to 18.0 and 7.1 percentage points over base models, and reducing distractor overlap in modality-specific captioning.