World models stop guessing whole frames: learn an uncertainty map, let diffusion fill only what it must

Future-frame prediction is split in two: deterministic change caused by the action comes straight from a deterministic network, while stochastic change such as disocclusions and dynamic textures is generated by a diffusion model. Earlier approaches optimized one side at the expense of the other; 3D Gaussian splatting applies only geometric warping and leaves information-less gaps in previously occluded regions, while conditional generative models synthesize the whole frame without explicitly identifying which changes are deterministic. On two-view extrapolation over the DL3DV benchmark, D3WM beats DepthSplat on PSNR at every horizon of 3, 6, 9 and 12 frames, the gap widening with horizon, and leads on PSNR, SSIM and LPIPS at 9 and 12 frames.

The uncertainty map is not a hand-labeled mask but a spatial variance the network regresses itself, serving both as conditioning for the diffusion model and as a spatial gate at inference. The deterministic network is trained with a Gaussian negative-log-likelihood objective, so it is forced to output high variance where its deterministic prediction error is high, marking the regions that genuinely need generation. At inference a per-pixel trust map blends the diffusion model's noisy latent with a forward-noised version of the deterministic prediction, collapsing to the deterministic prediction where uncertainty is low and to the diffusion output where it is high; ablations show that removing the inference gate or switching to a clean-state blend loses accuracy.

For robotic manipulation, feeding frames predicted by D3WM into a trajectory-generating policy yields trajectories closer to those from real frames than feeding iVideoGPT predictions. iVideoGPT distorts and hallucinates geometry on Push-T, producing physically impossible gripper and object locations and deforming the rigid T object being pushed. On 100 randomly selected frames from each of the 14 test-split episodes of Real PushT, predicting 16 frames ahead, D3WM reaches ADE 12.864 and FDE 17.550 against iVideoGPT's 45.134 and 65.228, with ground truth at 1.552 and 1.319.

The report covers action-conditioned single-frame future prediction: robotic tabletop manipulation and camera motion as two action spaces, plus long-horizon extrapolation from two-view inputs. Evaluation predicts 3, 6, 9 and 12 frames ahead on DL3DV, extrapolates out of distribution on RealEstate10k without retraining, and measures downstream trajectories and goal-image candidate verification on Push-T and BridgeData V2. The authors propose to experiment with varying reference view counts and to extend the targeted-generation framework beyond RGB images.

The authors note that the model produces only a single frame at each query, which is efficient when only the trajectory endpoint matters but limiting when assessing the viability of intermediate states, and that the formulation assumes scene changes are driven entirely by the input action sequence, so it does not account for predictable agent-independent dynamics. On BridgeData V2, IRASim remains stronger on policy consistency, which the authors attribute to IRASim diffusing the whole trajectory and enforcing consistency across the horizon, whereas D3WM, like WorldGym, predicts a single frame with no mechanism tying independently predicted futures together. The authors also state that the deterministic-only variant's higher PSNR comes from averaging in high-uncertainty regions and looking blurred, while the full model samples sharp outcomes at a PSNR cost, and whether the sharpness is worth that cost is not decidable from the table alone.