# 3D Understanding

4 related articles

Subscribe to updates

arXiv Nov 2, 2026
Same object, new camera angle, wrong answer: LieVLM predicts local rotations from geometry tokens and lifts ScanRefer grounding accuracy by 6.9 points
Multi-view 3D vision-language models suffer viewpoint-induced semantic drift: the same object and query produce weaker, diffuse or shifted target responses when the camera moves. LieVLM has geometry tokens predict local rotation parameters and channel-wise gates, applies length-preserving rotations to projected visual features within learned three-dimensional subspaces, and gates the identity-relative correction back into the original representation. On ScanRefer, LieVLM-8B reaches 64.5 Acc@0.25 and 57.2 Acc@0.5, 6.9 and 6.3 points above the published VG LLM-8B under the same proposal-refinement protocol.
Multi-view 3D vision-language models suffer viewpoint-induced semantic drift: the same object and query produce weaker, diffuse or shifted target responses when the camera moves. LieVLM has geometry tokens predict local rotation parameters and channel-wise gates, applies length-preserving rotations to projected visual features within learned three-dimensional subspaces, and gates the identity-relative correction back into the original representation. On ScanRefer, LieVLM-8B reaches 64.5 Acc@0.25 and 57.2 Acc@0.5, 6.9 and 6.3 points above the published VG LLM-8B under the same proposal-refinement protocol.
Tencent (China)University of Amsterdam Westlake University
Jingjun Yi（First author）
LieVLM Lie Routing 3D Understanding Spatial Reasoning Multimodal Large Language Models
arXiv Oct 2, 2026
OctLLM treats octree occupancy bytes as an explicit 3D language, improving 3D generation and understanding in one unified multimodal model while preserving language ability
OctLLM converts meshes into coordinate- and depth-anchored Sparse Octree (S-Octree) occupancy bytes as an explicit 3D language and places 3D capacity in full-rank branches separate from the pretrained weights, training about 2.80B parameters; it achieves the best 3D generation and understanding among unified multimodal LLMs, lowering image-to-3D Inception FID from ShapeLLM-Omni's 38.21 to 31.58 and raising render-grounded captioning from 18.14 to 46.88, while exactly reproducing the backbone on MMLU and HellaSwag and staying within one point on GSM8K.
OctLLM converts meshes into coordinate- and depth-anchored Sparse Octree (S-Octree) occupancy bytes as an explicit 3D language and places 3D capacity in full-rank branches separate from the pretrained weights, training about 2.80B parameters; it achieves the best 3D generation and understanding among unified multimodal LLMs, lowering image-to-3D Inception FID from ShapeLLM-Omni's 38.21 to 31.58 and raising render-grounded captioning from 18.14 to 46.88, while exactly reproducing the backbone on MMLU and HellaSwag and staying within one point on GSM8K.
Ran Dan（First author）
OctLLM Sparse Octree (S-Octree)Token-Routed Dual-Stream Transformer 3D Generation 3D Understanding
arXiv Jul 8, 2026
ELSA3D unifies 3D understanding and generation via elastic semantic anchoring, reaching state-of-the-art image-to-3D, text-to-3D, and 3D captioning while roughly halving FLOPs and inference latency
ELSA3D introduces elastic semantic anchoring, representing geometry with a scale-aware octree tokenizer and adding sparse cross-modal Anchor Tokens that select semantic cues, route them to the most relevant 3D scale, and write the fused signal back into the unified representation, with a lightweight per-block router making computation and reasoning elastic; it achieves state-of-the-art performance across image-to-3D generation, text-to-3D generation, and 3D captioning, outperforming the strongest unified baseline while roughly halving FLOPs and inference latency relative to the non-elastic version of the same model.
ELSA3D introduces elastic semantic anchoring, representing geometry with a scale-aware octree tokenizer and adding sparse cross-modal Anchor Tokens that select semantic cues, route them to the most relevant 3D scale, and write the fused signal back into the unified representation, with a lightweight per-block router making computation and reasoning elastic; it achieves state-of-the-art performance across image-to-3D generation, text-to-3D generation, and 3D captioning, outperforming the strongest unified baseline while roughly halving FLOPs and inference latency relative to the non-elastic version of the same model.
Tianjiao Yu（First author）
ELSA3D Elastic Semantic Anchoring Image-to-3D Generation Text-to-3D Generation 3D Understanding
arXiv Oct 1, 2026
Imagine3D-LLM has multimodal LLMs imagine a compact 3D scene before answering, reaching 68.5 average on SPAR-Bench at 7B scale
The work introduces Imagine3D-LLM: a small set of learnable Gaussian summary tokens is appended after the image tokens, decoded into a compact 3D Gaussian Splatting representation, and trained jointly with a photometric reconstruction loss and standard next-token prediction, so that a multi-view multimodal LLM assembles a coarse 3D scene representation before answering; it consistently outperforms prior approaches across seven spatial reasoning and 3D scene understanding benchmarks, reaching 68.5 average on SPAR-Bench and surpassing the strongest spatially-aware baseline 3DThinker-7B by 5.2 points.
The work introduces Imagine3D-LLM: a small set of learnable Gaussian summary tokens is appended after the image tokens, decoded into a compact 3D Gaussian Splatting representation, and trained jointly with a photometric reconstruction loss and standard next-token prediction, so that a multi-view multimodal LLM assembles a coarse 3D scene representation before answering; it consistently outperforms prior approaches across seven spatial reasoning and 3D scene understanding benchmarks, reaching 68.5 average on SPAR-Bench and surpassing the strongest spatially-aware baseline 3DThinker-7B by 5.2 points.
Imagine3D-LLM Gaussian Splatting Spatial Reasoning Multimodal Large Language Models 3D Understanding

You’re all caught up

Same object, new camera angle, wrong answer: LieVLM predicts local rotations from geometry tokens and lifts ScanRefer grounding accuracy by 6.9 points

OctLLM treats octree occupancy bytes as an explicit 3D language, improving 3D generation and understanding in one unified multimodal model while preserving language ability

ELSA3D unifies 3D understanding and generation via elastic semantic anchoring, reaching state-of-the-art image-to-3D, text-to-3D, and 3D captioning while roughly halving FLOPs and inference latency

Imagine3D-LLM has multimodal LLMs imagine a compact 3D scene before answering, reaching 68.5 average on SPAR-Bench at 7B scale