Visual tokens need not be ranked alike: BtsPruner keeps 32 of 576 and retains 91.6% of the dense model's performance
Synopsis
Large vision-language models process hundreds to thousands of visual tokens per image, and inference cost rises with them. Earlier training-free pruning methods ranked all visual tokens by one importance criterion, overlooking that tokens take on different roles as encoding proceeds. BtsPruner locates the role transition from changes in attention concentration, then picks globally referenced representative tokens from later layers and recursively retrieves local detail tokens from earlier layers, preserving both roles under a fixed budget; on LLaVA-1.5-7B it keeps only 32 of 576 visual tokens (5.6%) and retains 91.6% of the dense model's performance.
Interpretation
Visual token pruning no longer applies one importance ranking to every token; it keeps two roles that form during encoding, tokens carrying local detail and tokens referenced globally. Earlier training-free methods, including FastV, which ranks tokens by text-to-visual attention inside the language model, and methods near the vision encoder using visual attention, feature similarity or their combination, assumed visual tokens could be ranked by a unified criterion. On LLaVA-1.5-7B with 32 retained visual tokens (5.6% of 576), BtsPruner retains 91.6% of the dense model's performance, 2.5 percentage points above the strongest compared method, VisPruner, in relative retention; at 128 and 64 retained tokens the margins are 0.9 and 1.1 points.
The method first finds the encoding layer where token roles turn over, then draws the two kinds of tokens from after and before that point. The transition point is set by the layer-to-layer change in [CLS] attention concentration, with a threshold coefficient defaulting to 2; encoders without a [CLS] token use the mean incoming attention of visual tokens instead. Global representative candidates are selected by [CLS] attention at the second-to-last layer and filtered for diversity so the kept tokens differ from one another; local detail tokens start from the transition layer, take each selected representative as a parent, draw candidates by attention and expand recursively until the token budget or the earliest layer is reached.
In the high-resolution LLaVA-NeXT-7B setting, keeping only 160 visual tokens cuts CUDA latency from 821 ms to 138 ms. That model originally processes 2880 visual tokens, so pruning weighs more heavily on inference cost in high-resolution scenarios. In the same setting, 640 retained tokens preserve 99.1% of the dense model's performance and 160 retained tokens preserve 91.3%; the efficiency comparison runs on a single RTX 3090 with the POPE benchmark.
Perspective
The demonstrated scope is training-free inference-time pruning: on LLaVA-1.5-7B, LLaVA-NeXT-7B, Qwen-VL-7B, InternVL-13B and CogVLM-7B, run on ten multimodal benchmarks under each model's original evaluation settings; the efficiency figures are FLOPs, cache, GPU memory and CUDA latency measured on a single RTX 3090 with the POPE benchmark.
The role evolution is observed from attention statistics of the LLaVA-1.5-7B vision encoder on one representative VQAv2 example, and attention weights do not give a complete causal explanation of representation formation, since token representations are also shaped by value projections, residual connections and feed-forward networks. Qwen-VL's query-based projector requires reselecting query tokens by their attention over retained tokens, a lightweight adaptation rather than a direct application.