ELSA3D unifies 3D understanding and generation via elastic semantic anchoring, reaching state-of-the-art image-to-3D, text-to-3D, and 3D captioning while roughly halving FLOPs and inference latency
Related research and updates
Synopsis
ELSA3D introduces elastic semantic anchoring, representing geometry with a scale-aware octree tokenizer and adding sparse cross-modal Anchor Tokens that select semantic cues, route them to the most relevant 3D scale, and write the fused signal back into the unified representation, with a lightweight per-block router making computation and reasoning elastic; it achieves state-of-the-art performance across image-to-3D generation, text-to-3D generation, and 3D captioning, outperforming the strongest unified baseline while roughly halving FLOPs and inference latency relative to the non-elastic version of the same model.
Figure 1: ELSA3D overview. ELSA3D is built around elastic semantic anchoring , where routing jointly controls computation and semantic–geometric grounding. (i) The router has three heads: a Gating Head ( p i p^{i} , skip or run), a Width Head ( q i q^{i} , MLP width), and an Anchor Routing Head ( β i , α i \beta^{i},\alpha^{i} , which text tokens become anchors and at which scale). (ii) Blocks with p i ≥ τ p^{i}\geq\tau execute at the selected width; others are skipped. (iii) Selected text tokens are routed to their preferred scale, cross-attended to the 3D tokens at that scale, and fused into the anchor set.
arXivInterpretation
Introduces elastic semantic anchoring, structuring language and geometric reasoning jointly along matched abstraction scales instead of concatenating text and 3D tokens into a flat sequence and relying on self-attention for implicit interaction. In existing unified 3D foundation models text-3D interaction remains largely implicit, and the flat sequence collapses coarse structural cues and fine geometric details into one undifferentiated representation; this work organizes cross-modal interaction along matched abstraction scales. The abstract presents the method design and motivation contrast, and reports state-of-the-art performance across image-to-3D generation, text-to-3D generation, and 3D captioning.
Introduces Anchor Tokens, sparse cross-modal units that select semantic cues, route them to the most relevant 3D scale, retrieve scale-specific geometric evidence, and write the fused signal back into the unified representation. Cross-modal interaction shifts from dense self-attention to sparse yet precise anchoring, with semantic cues explicitly assigned to corresponding geometric scales. The abstract describes the four steps of selection, routing, retrieval, and write-back, and states that interaction stays sparse yet precise.
Represents geometry with a scale-aware octree tokenizer, providing the source of scale-specific geometric evidence for cross-scale anchoring. The geometric representation itself carries scale structure, so semantic cues can be retrieved and fused per scale rather than mixing coarse and fine detail in a single representation. The abstract states the tokenizer is a scale-aware octree structure and serves as the basis for Anchor Tokens retrieving scale-specific evidence.
A lightweight per-block router decides which text tokens instantiate anchors at which geometric scale, making both computation and reasoning elastic so cross-modal capacity concentrates where alignment is most needed. Elastic routing allocates cross-modal capacity on demand rather than computing uniformly over all tokens and scales. The abstract reports roughly halved FLOPs and inference latency relative to the non-elastic version of the same model, while performance exceeds the strongest unified baseline.
Perspective
The work targets unified 3D foundation model settings, applicable to tasks that require both 3D asset generation and language reasoning within a single backbone, specifically covering image-to-3D generation, text-to-3D generation, and 3D captioning. Its elastic semantic anchoring and per-block routing target cross-scale alignment between text and geometry, suited to settings where geometry has multi-scale structure and cross-modal capacity should be allocated on demand. For researchers and practitioners seeking to reduce the inference cost of unified models, the design offers a path that replaces dense cross-modal attention with sparse anchoring.
The visible text is abstract-level information: it gives no specific metric values per task, dataset and evaluation settings, or baseline configurations, and no ablations separating the contributions of Anchor Tokens, the scale-aware tokenizer, and the per-block router. The roughly halved FLOPs and inference latency are measured against the non-elastic version of the same model, and behavior across hardware and batch sizes still needs the original data. Design choices such as the number of scales, anchor sparsity, and the interpretability of routing decisions, along with the method's applicability boundary on broader 3D tasks, remain open questions worth watching.