Video LLMs can compress tokens without comparing every pair: codec motion and residuals cut temporal matching by 64.7%

Visual token compression for video large language models gains a training-free cue: compressed video bitstreams already carry reference-frame dependencies, motion vectors, and residuals, and these coding signals can serve as a low-cost search prior that narrows what a compression decision has to compare. Earlier training-free compression methods mainly target the number of tokens sent to the language model, while the compression decision itself still requires many visual feature comparisons, especially for intra-frame token protection and inter-frame temporal merging. Across LLaVA-OneVision, LLaVA-Video, and Qwen2.5-VL-7B-Instruct on Video-MME, EgoSchema, LongVideoBench, and MVBench, average scores match the training-free baseline FlashVID and exceed it in some settings; on Video-MME, temporal comparisons drop 64.7% relative to FlashVID, and token-protection comparisons drop 79.8% at a 25% retention ratio and 96.6% at 10%.

The method first aligns coding information to the visual token grid, then gives residuals and motion separate jobs: residuals bound the intra-frame candidates worth protecting, motion builds cross-frame candidates between sparsely sampled frames, and local visual feature similarity still makes the final keep-or-merge call. Coding blocks and visual tokens are misaligned in space and time, since one token may cover several coding blocks and the coding reference of a current sampled frame is often an intermediate frame rather than the preceding sampled frame, so a single-step motion vector cannot directly give token correspondence between two sampled frames. Motion is propagated along the reference-frame chain of the complete video, composing several short coding dependencies into a sampled-frame-level position, and visual similarity is then evaluated inside a local window of radius 2 around that predicted position; abnormal propagation relaxes the radius to 4, and too few candidates fall back to global matching, all without modifying the base VLLM or retraining the vision encoder or language model.

The work targets video large language model inference that consumes compressed video bitstreams, and its readers are researchers and engineers concerned with multimodal inference efficiency. The method retains reference-frame relations, motion vectors, and residuals during decoding, aligns them to the visual token grid, and uses them for token compression while leaving the base VLLM, vision encoder, and language model unchanged. Experiments cover LLaVA-OneVision, LLaVA-Video, and Qwen2.5-VL-7B-Instruct across Video-MME, EgoSchema, LongVideoBench, and MVBench, with the comparison-count measurements taken on Video-MME with LLaVA-OneVision.

Coding information does not directly decide which tokens are kept or merged: large residuals may also come from texture, compression noise, or coding errors, so residuals only build a candidate pool and the final selection still uses visual feature attention. Reference-chain propagation can meet missing or abnormal motion information, in which case it first searches nearby token cells within a radius of 1 and otherwise reuses the motion vector from the previous step; abnormal propagation or too few local candidates switches to a relaxed radius or a global fallback. Raising the local search radius from 2 to 4 improves the overall score by only 0.1 while increasing the maximum candidates for an interior token from 25 to 81, so 2 is the default.