Feed-forward 3D reconstruction stops being final: Click3R lets a user fix geometry with a few correspondence clicks

Feed-forward stereo 3D reconstruction moves from a single irrevocable prediction to interactive correction: a user clicks a few corresponding points in the two views, and the model re-emits dense 3D pointmaps under those constraints. Optimization-based reconstruction can iteratively refine its estimate, whereas feed-forward models such as DUSt3R emit one output and fail silently on repetitive textures, symmetric structures or large viewpoint changes; test-time optimization is computationally expensive and cannot resolve genuinely ambiguous scenes at inference. Across the four Hard3DBench conditions, 2-view and 5-view, in-distribution and out-of-distribution, Click3R improves mAA@30° over DUSt3R retrained on the same data, with the largest gain on the 5-view in-distribution split, and pointmap error also falls in all four without higher input resolution or more training data.

The correction comes from a correspondence adapter placed between encoder and decoder that turns clicks into prompt tokens and exchanges them with image tokens through two-way cross-attention. The adapter follows SAM-style two-way attention: prompt tokens self-attend, then cross-attend with image tokens in both directions, and image positional encodings share one coordinate space with the prompts so sparse clicks and dense features are compared at the same scale. Training prompts are generated by projecting ground-truth 3D geometry, with 30% of samples receiving no prompts, 40% receiving 1 to 4 and 30% receiving 5 to 12; with no prompts the adapter reduces to an identity mapping and the model behaves like the base DUSt3R, while with prompts the forward pass adds about 5% GPU compute on a single A100.

Human clicks come close to perfect correspondences, so sparse, imperfect human input already captures most of the benefit. Earlier experiments used ground-truth correspondences derived from known 3D geometry and did not test realistic annotation conditions. One annotator unfamiliar with the datasets freely clicked on 100 image pairs from the Hard3DBench out-of-distribution set, averaging 3.8 correspondences per pair over a range of 1 to 7; against a random ground-truth oracle, the human labels recovered about 96% of the median rotation error improvement.

The demonstrated scope is two-view stereo reconstruction, with the training resolution not stated numerically in the material, and multi-view capability shown indirectly through global alignment; evaluation covers the CO3D, ScanNet++ and ARKitScenes standard benchmarks plus the 2-view and 5-view, in-distribution and out-of-distribution splits of the curated Hard3DBench and 100 human-labeled pairs. The intended readers are researchers and engineers working on 3D reconstruction and multi-view geometry.

Clicks must be co-visible in both views, so the two inputs need a shared visible region; completely incorrect clicks degrade pose accuracy monotonically, and all-incorrect clicks fall below using no prompts. The adapter injects correspondences as soft feature conditioning rather than hard geometric constraints, and the signal is progressively diluted across twelve subsequent decoder cross-attention blocks; on nearly non-overlapping views, extreme rotations, large outdoor scenes and textureless surfaces, reconstruction still distorts or improves only marginally.