3D reconstruction drops dense attention layer by layer: up to 5.6x end-to-end speedup at 1,024 views

Pretrained feed-forward 3D reconstruction models no longer need every global-attention layer to run full softmax over all token pairs; some layers can use linear attention while others keep exact softmax over selected spatial tiles. Earlier acceleration replaced the whole attention stack with one approximation, such as uniform sparsity, uniform linear attention, or key/value subsampling, without distinguishing how tolerant each layer is. Across VGGT, HunyuanWorld-Mirror, and Pi3, sensitivity-guided allocation gives the highest RealEstate10K AUC@30 at a fixed 0.5L/0.8S budget, 0.730, 0.850, and 0.866 respectively, while uniform, reversed, and random assignments fall well below.

The converted model is usable immediately without gradient updates, and an optional short refinement recovers accuracy. Training-free conversion needs only a closed-form fit, whereas earlier sparse or linear replacements typically required retraining or distillation to recover accuracy. The training-free 0.5L/0.6S setting stays within 0.7, 0.4, and 0.6 percentage points of dense VGGT, HYWM, and Pi3 on RealEstate10K AUC@30 at about 79% mean attention sparsity; after 3,000 refinement steps, Zebra3D-VGGT reaches 80.5% AUC@30 at about 94% sparsity, within 1.1 points of dense VGGT fine-tuned on the same data and schedule.

Long sequences gain the most, because global attention takes a larger share of runtime as view count grows. At 100 views the speedup is modest, while at 1,000 views global attention dominates the dense model's cost. On ScanNet-50 at 1,000 frames, Zebra3D-KD (0.7L/0.8S) reports a Chamfer distance of 0.460 at 5.60x speedup, against 0.466 at 1.00x for dense VGGT; at 1,024 views the three backbones reach end-to-end speedups on the order of 5.6x, 4.6x, and 5.6x.

The method targets already-trained feed-forward 3D reconstruction backbones, replacing their global-attention layers with a linear and tile-sparse hybrid; conversion itself needs no gradient updates, while optional refinement requires one extra frozen-teacher forward pass. Evaluation covers reconstruction on 7Scenes, NRGBD, and DTU, camera pose on RealEstate10K and CO3Dv2, and long sequences of 100 to 1,000 frames on ScanNet-50, with wide-baseline checks on Tanks and Temples and ETH3D. The authors list input-adaptive allocation and broader hardware support as future work.

The authors note that gains mainly appear on long sequences, with modest speedups at 8 to 16 views because the image encoder and prediction heads increasingly bound end-to-end acceleration. The stage allocation is fixed per backbone and derived from marginal single-stage sensitivity, so it does not model interactions among simultaneously converted stages or adapt to individual inputs. The allocation is stable across the tested calibration datasets, but broader generalization to substantially different architectures and domains remains to be established. The optimized kernels and benchmarks target Triton on AMD MI355X GPUs, and portability and performance on other accelerators require further evaluation. Optional distillation needs a frozen dense teacher during refinement, though it adds no inference-time cost.