DexForesight stops dexterous hands from imagining the future: spatial foresight moves into policy latents, reaching 58.3% on real robots
Synopsis
DexForesight injects predictive features from an action-conditioned spatial world model into a dexterous policy's latent space through query alignment, and at deployment keeps only the policy, which acts from RGB, language and robot state. On 11 DexJoCo simulation tasks it raises the GR00T N1.7 backbone's overall success from 49.4% to 60.5%, and on three real-robot tasks it reaches 58.3%, 8.3 points above the strongest baseline. The world model, depth encoder and future-prediction heads are all removed at inference.
Interpretation
A dexterous manipulation policy can acquire spatial foresight over hand and scene without generating any future frames, relying at deployment only on current RGB images, language instructions and robot state. Earlier world-action models typically make future prediction part of inference, generating future images, motion or spatial states at each control step and then using them to select actions. Across 11 DexJoCo simulation tasks, DexForesight reaches 60.5% overall success, 9.6 points above the strongest baseline Fast-WAM and 11.1 points above its own GR00T N1.7 backbone, while running neither the spatial world model nor future decoding at deployment.
The route is to first train an action-conditioned spatial world model during training, then align its predictive features into the policy's query representations. That world model takes offline-estimated scene depth, hand geometry computed from robot states through forward kinematics, and demonstration trajectories as input, and predicts 3D displacements of fixed hand points plus changes in image features. The world model is trained on roughly 370 hours of demonstrations spanning 60 tasks in five categories, with a 100-hour subset carrying depth estimated offline by StereoAnyVideo; it is then frozen and its query features are transferred to the policy through query alignment, which conditions the action expert.
Perspective
The demonstrated scope is 11 DexJoCo simulation tasks and three real-robot tasks (Close Box, Insert Battery, Screw On Lid), with 20 trials per real-robot task, all using held-out object positions, and Screw also varying cup color. The intended readers are researchers and engineers working on dexterous manipulation and vision-language-action models.
On the real robot, WLA and Fast-WAM are external comparisons to alternative train-deploy strategies, and architectural and training differences preclude treating either as a one-variable causal control. The latent probe shows DexForesight's error below the architecture-matched QwenVLA at all three horizons of 0.13, 0.27 and 0.54 s, but probe error itself is not the same as task success rate.