Video language models lose count of repeated actions: answers stick to a few favorite numbers while the count stays in the last layer
Synopsis
RepProbe turns repetition count into a controlled variable and finds that generalist video language models collapse at high counts even though the count remains linearly decodable from the same final-layer state, locating the failure in the output pathway rather than the representation.
Interpretation
Generalist video language models miscount badly as repetitions accumulate, while specialist counters stay accurate. Earlier counting evaluations read only the final answer, and count was entangled with clip duration and source, so error could not be attributed to the count itself. On the 32K real clips of RepProbe-Real, the specialist ESCounts retains 65.6% off-by-one accuracy in the Many band, while every generalist reaches at most 22.2% there.
The count is not missing from the model; it is simply not reported. Behavioral scores alone could not separate a missing count from a count that is present but lost on the way to the answer. Read from the same final-layer state, a linear probe lowers MAE from 3.91 to 2.49 for Cosmos-Reason2 and from 7.88 to 3.05 for Qwen3-VL, with both readouts taken from the same forward pass on the same frames.
Generated answers concentrate on a small set of preferred numbers instead of tracking the true count. Previously only the growth of error with count was known, not the shape of the wrong answers. Under a length-fair readout over the single-token number words one through twenty, Qwen3-VL settles on fifteen and Cosmos-Reason2 on eighteen, and how often each number appears tracks its lexical frequency in text.
Perspective
The work addresses researchers studying temporal reasoning in video language models, in settings where the repetition count is known and exactly controlled, covering real exercise clips and procedurally generated periodic stimuli. The probing analysis focuses on the two strongest generalists, Cosmos-Reason2 and Qwen3-VL, and the authors list extending it to more models and other temporal quantities as a natural next step.
The authors note that the synthetic part isolates periodicity by design and that the real-video component is limited to exercise footage, so broadening it would further test generality. The output-side failure is characterized as answers concentrating on preferred numbers, and the mechanism producing those preferences is left to future work.