Same object, new camera angle, wrong answer: LieVLM predicts local rotations from geometry tokens and lifts ScanRefer grounding accuracy by 6.9 points
Synopsis
Multi-view 3D vision-language models suffer viewpoint-induced semantic drift: the same object and query produce weaker, diffuse or shifted target responses when the camera moves. LieVLM has geometry tokens predict local rotation parameters and channel-wise gates, applies length-preserving rotations to projected visual features within learned three-dimensional subspaces, and gates the identity-relative correction back into the original representation. On ScanRefer, LieVLM-8B reaches 64.5 Acc@0.25 and 57.2 Acc@0.5, 6.9 and 6.3 points above the published VG LLM-8B under the same proposal-refinement protocol.
Interpretation
A 3D vision-language model can give different target-related responses to the same scene, target and query as the camera viewpoint changes, with target activations weakening, becoming diffuse or shifting to irrelevant regions, a phenomenon called viewpoint-induced semantic drift. Earlier 3D multimodal models mainly treated geometry features as auxiliary context, injecting them additively or through attention, without explicitly defining how geometry should drive semantic changes across viewpoints. In a controlled comparison that fixes the scene, target, query, checkpoint and reference view and varies only the viewpoint along one camera trajectory, responses without routing drift markedly across near, medium and far views, while Lie routing keeps target responses more localized.
LieVLM uses geometry tokens to predict, for each visual token and each three-dimensional channel group, a local rotation generator and a channel-wise routing coefficient, and writes back only the correction relative to the identity transform. A bias-free output projection maps the correction back to the token space, where it is fused element-wise with the original 2D semantic path and a direct geometry path under the predicted gate, so pretrained semantics and direct spatial context are both retained. On ScanRefer, Scan2Cap and EmbodiedScan 3D video object detection, against VG LLM baselines that share backbone, data and training and differ only in additive fusion versus Lie routing, LieVLM improves consistently across training scales and a stronger backbone.
On Scan2Cap 3D dense captioning, LieVLM-8B reaches a CIDEr of 82.7, above Video-3D LLM at 80.0 and LLaVA-3D at 79.2. Against the RGB-based geometry-injection baseline VG LLM-8B, CIDEr rises from 80.0 to 82.7 while the other captioning metrics stay comparable. On EmbodiedScan 3D video object detection, F1 rises from 41.2 for VG LLM-8B to 45.8 in the 4-frame setting and from 40.0 to 44.1 in the 6-frame setting, and three-seed paired runs are positive on all six aggregate precision, recall and F1 metrics.
Perspective
Evaluation centers on static indoor benchmarks such as ScanRefer, Scan2Cap and EmbodiedScan, plus the spatial reasoning benchmarks SPAR-Bench, CV-Bench-3D, BLINK-Spatial and VSI-Bench; dynamic objects, outdoor environments and long-horizon trajectories are left for future study.
The routing gate and Padé normalization only constrain moderate geometry perturbations, and severe geometry degradation can still reduce the reliability of routed corrections; the rotation constraint applies to the learned three-dimensional channel groups and does not impose equivariance or composition consistency with physical camera transformations on the complete model.