Audio language models miss much of the recording: adding acoustic training lifts structure-task accuracy by up to 44.4 points

Audio language model benchmark scores confound language modeling ability with acoustic understanding, because many items can be answered from the transcript or even the question alone. There was no reference that keeps the original words while replacing only the voice, so the accuracy the original recording adds beyond text could not be estimated separately. Across 92 tasks and 13,707 items from 6 benchmarks, the original recording is compared with synthetic readings of the same transcript and with the transcript added alongside it, yielding two recording gains.

The 19 open models fall clearly behind Gemini 3.1 Pro on tasks needing global attributes or within-recording structure, and about half or more of the gap comes from their smaller recording gains. The open-model mean is 46.7% on attribute tasks and 33.5% on structure tasks, against 75.3% and 75.4% for Gemini 3.1 Pro, gaps of 28.6 and 41.9 percentage points. The two recording gains differ between Gemini and the open-model mean by 17.9/15.8 and 23.3/19.9 points, in the same direction.

In Gemma-4 12B and Qwen2.5-Omni-7B, the acoustic information needed for answers is readable by linear probes in early language-model layers, but readout declines in later layers and answer accuracy stays well below it. For pitch, readout reaches 97.9% at layer 4 of Gemma-4 12B and 88.0% at layer 2 of Qwen2.5-Omni-7B, while answer accuracy is 49.4% and 54.3% against a chance level of 50%. Peak readout exceeds answer accuracy on every probed task in both models, and for most tasks the peak lies in the early layers.

Training LoRA adapters with a content anchor in the readout-decline layers substantially raises attribute and structure accuracy while keeping lexical accuracy near the base model. Gemma-4 12B rises from 34.7% to 53.6% on attribute tasks and from 35.4% to 66.2% on structure tasks; Qwen2.5-Omni-7B rises from 56.7% to 69.5% and from 34.0% to 78.4%. The gains in the two models range from 12.8 to 44.4 points, mean lexical accuracy stays within 1 point of the base models, and most of the improvement appears as larger recording gains.

Measurement and repair cover 92 automatically scorable tasks and 13,707 items from 6 benchmarks, and the repair is carried out only on Gemma-4 12B and Qwen2.5-Omni-7B; most items ask one question about one recording, and multi-party dialogue tasks form only a small part of the evaluation.

Both recording gains depend on the same Whisper transcript, so recognition errors affect them together; the probes are trained with supervision and receive segment boundaries on structure items, so the gap between readout and answer accuracy also reflects these advantages and does not by itself show that the models leave available information unused.