Multi-view 3D scenes no longer need every layer: GeoGR keeps over 95% performance at a tenth of the compute
Synopsis
Multi-view 3D vision-language models split a scene into 32 views and produce more than six thousand visual tokens, making inference expensive. GeoGR compresses at two levels: at the projector it consolidates cross-view observations into voxel tokens selected by semantic saliency and spatial coverage, merging unselected voxel features into contextual anchors; inside the LLM it computes visual tokens only within a middle-layer window, letting low-relevance tokens skip whole layers with hidden states preserved and rejoin at anchor layers. On LLaVA-OneVision-7B and Video-3D LLM-7B it retains 95.2% and 96.8% of the dense models' performance, with clear LLM prefill and time-to-first-token speedups.
Interpretation
The visual token sequence of a multi-view 3D scene is compressed to roughly a tenth of the compute budget while most task performance is kept. Previously, multi-view 3D vision-language models used 32 views per scene, yielding 6,272 visual tokens, and processed all visual tokens at every LLM layer. It retains 95.18% of the dense model's average performance on LLaVA-OneVision-7B and 96.80% on Video-3D LLM-7B, both measured at about 10% of LLM-side visual FLOPs.
Compression happens at two levels: the projector merges repeated cross-view observations, and the LLM allocates computation by query relevance. Projector-level methods previously selected either by saliency alone, discarding background regions, or by geometric diversity alone; LLM-level methods previously pruned tokens irreversibly. GeoSemZip back-projects cross-view observations into a shared world coordinate system and aggregates them into voxel tokens, keeping high-saliency dominant tokens and sampling contextual anchors along 3D Morton order, then merging unselected voxel features into anchors by cosine similarity; GroupRoute uses late entry and early exit to define a visual computation window, groups tokens at anchor layers by query attention, lets the low-priority group bypass later layers with hidden states preserved, and rescores at the next anchor.
Low-relevance visual tokens are treated as temporarily inactive rather than permanently redundant. At larger layer distances, initially low-ranked tokens enter the top half and take a non-negligible share of its attention, so a single-layer ranking cannot justify permanent pruning. Tracking layers 8 to 23 on ScanQA with LLaVA-OneVision shows higher ranking agreement between nearby layers that weakens as layer distance grows; at a matched average token budget, recoverable skipping at layers 9 and 13 retains 95.18% on average, above single-anchor pruning at 94.65% and 95.03% and two-anchor pruning at 94.73%.
Perspective
Evaluation covers two multi-view 3D vision-language models: LLaVA-OneVision-7B, which has no explicit 3D spatial encoding and is evaluated only on two question-answering benchmarks, and Video-3D LLM-7B, which carries depth- and pose-derived 3D positional encoding and is evaluated on five ScanNet benchmarks spanning question answering, visual grounding and dense captioning. Compression-aware post-training uses 67K samples selected by PRISM from 223K, retaining 95.69% and 98.72% of full-data post-trained performance for the two models. The method needs calibrated views, aligned depth and camera poses to build voxels.
The cross-layer token relevance analysis is done only on ScanQA with LLaVA-OneVision, anchor positions are chosen by BARS under a fixed budget and fixed at inference while token groups remain query-dependent, and compression-aware post-training runs in two stages with the vision encoder frozen in both. Efficiency gains depend on projector overhead, so a method with a costly projector stage can offset its prefill savings.