Video LLMs stop comparing every token: codec motion and residuals cut matching by two-thirds

For video large language models, the cost of compressing visual tokens includes the comparisons needed to make the compression decisions, and this method lowers that cost. Existing training-free compression methods mainly reduce the number of tokens sent to the language model, but intra-frame token protection and inter-frame temporal merging still require many visual feature comparisons. On Video-MME with LLaVA-OneVision as the base model, temporal comparisons relative to FlashVID fall by 64.7% at all four retention ratios, and selection comparisons fall from 79.8% at 25% retention to 96.6% at 10% retention.

Compressed video already carries reference-frame dependencies, motion vectors and residuals from decoding, and these coding signals serve as a low-cost search prior for candidate construction. Coding blocks and visual tokens differ in temporal and spatial granularity, and one token may cover several coding blocks with different motion directions, so coding signals generate candidates rather than deciding retention or merging directly. Residuals are averaged as absolute values over each token region and combined with a positive temporal residual increase relative to adjacent sampled frames to form the protection pool; motion is propagated along the reference-frame chain of the complete video to recover correspondences between sparsely sampled frames, and visual similarity is computed only in a local window around the predicted position.

On three representative video LLMs and four video-understanding benchmarks, compressed average scores match the training-free baseline FlashVID and exceed it in some settings. The prior strong training-free baseline is FlashVID, which selects tokens by attention and diversity and then performs tree-based spatiotemporal merging. LLaVA-OneVision reaches the same average score of 58.7 at 25% retention for both methods; at 10% retention the new method scores 57.0 and retains 97.6% of the vanilla average, while the Video-MME long-video subset improves over FlashVID by 0.9, 0.6, 1.4 and 2.0 points at the four retention ratios.

The method targets video stored or transmitted as compressed bitstreams, retaining reference-frame relations, motion vectors and residuals during decoding and aligning them to the visual token grid; the base video LLM, vision encoder and language model are left unmodified and no additional training is performed. Evaluation covers three models, LLaVA-OneVision, LLaVA-Video and Qwen2.5-VL-7B-Instruct, and four benchmarks, Video-MME, EgoSchema, LongVideoBench and MVBench, with comparison reductions measured on Video-MME using LLaVA-OneVision. The local matching radius defaults to 2, the relaxed radius to 4, the temporal merging threshold to 0.75, the temporal gap threshold in reference-chain propagation to 4, and the positive temporal residual increase weight to 0.2.

Large residuals are not necessarily semantically important, since texture, compression noise or coding errors can produce similar responses, so residuals only build the candidate pool while visual feature similarity makes the final retention and merging decisions. Coding blocks and visual tokens differ in spatial granularity, and one token may cover several coding blocks with different reference directions and motion vectors, requiring reference-frame voting and motion aggregation first. Reference-chain propagation can meet missing or abnormal motion information: it searches nearby token cells within radius 1, reuses the previous motion vector if none is found, switches to a relaxed radius when propagation terminates abnormally or the candidate set is too small, and falls back to global matching if candidates remain insufficient. Increasing the local search radius to 4 improves the overall score by only 0.1 points over radius 2, while raising the maximum candidate count for an interior token from 25 to 81.