# Latent World Model

Pretrained latent world models are usually frozen at test time, and distribution shift can degrade planning with them; updating only on prediction loss fits the outcomes of the agent's own actions and can stay wrong about actions that would reach the goal. OpTTA adds the planner's goal-reaching cost on candidate imagined rollouts to the prediction loss, biasing adaptation toward model parameters that are optimistic about reaching the goal. Across three JEPA world models and 15 shifted settings, adapting only the predictor raises final success over the frozen model by 13.5 percentage points on average, versus 6.9 for prediction-only AdaJEPA; with the encoder also adapted, the gains are 13.5 versus 7.5 points.

Pretrained latent world models are usually frozen at test time, and distribution shift can degrade planning with them; updating only on prediction loss fits the outcomes of the agent's own actions and can stay wrong about actions that would reach the goal. OpTTA adds the planner's goal-reaching cost on candidate imagined rollouts to the prediction loss, biasing adaptation toward model parameters that are optimistic about reaching the goal. Across three JEPA world models and 15 shifted settings, adapting only the predictor raises final success over the frozen model by 13.5 percentage points on average, versus 6.9 for prediction-only AdaJEPA; with the encoder also adapted, the gains are 13.5 versus 7.5 points.

Particle image velocimetry (PIV) usually fixes the imaging position before the measurement, so if the structure of interest departs from expectation, no later algorithm can recover information that was never captured. This work treats camera placement as a closed-loop decision variable: a DreamerV3 agent trained in Isaac Sim drives a 6-DoF arm using only the velocity field and uncertainty estimated by LIMA to locate, frame and resolve a hydrofoil wake. After zero-shot transfer to a real water tunnel, the agent locates wakes of randomly placed hydrofoils and measures them at higher resolution and lower flow-statistics error than a fixed view, in roughly half the time of a full scan and with far fewer image pairs.

Particle image velocimetry (PIV) usually fixes the imaging position before the measurement, so if the structure of interest departs from expectation, no later algorithm can recover information that was never captured. This work treats camera placement as a closed-loop decision variable: a DreamerV3 agent trained in Isaac Sim drives a 6-DoF arm using only the velocity field and uncertainty estimated by LIMA to locate, frame and resolve a hydrofoil wake. After zero-shot transfer to a real water tunnel, the agent locates wakes of randomly placed hydrofoils and measures them at higher resolution and lower flow-statistics error than a fixed view, in roughly half the time of a full scan and with far fewer image pairs.

The authors introduce ATLAS, a training objective that transfers normalized pairwise structure from an encoder's mean-pooled patch features to the planning latent and uses Wasserstein embedding matching (WEMReg) based on one-dimensional Wasserstein-2 transport to calibrate its marginal; instantiated in LeWM, ATLAS improves mean goal-reaching success across PushT, TwoRoom, and OGBench-Cube on both lower- and higher-novelty evaluation subsets, with the largest gain on higher-novelty TwoRoom episodes, while diagnostics show stronger novelty-related structure in the planning latent, improved marginal calibration, and lower multi-step prediction error.

The authors introduce ATLAS, a training objective that transfers normalized pairwise structure from an encoder's mean-pooled patch features to the planning latent and uses Wasserstein embedding matching (WEMReg) based on one-dimensional Wasserstein-2 transport to calibrate its marginal; instantiated in LeWM, ATLAS improves mean goal-reaching success across PushT, TwoRoom, and OGBench-Cube on both lower- and higher-novelty evaluation subsets, with the largest gain on higher-novelty TwoRoom episodes, while diagnostics show stronger novelty-related structure in the planning latent, improved marginal calibration, and lower multi-step prediction error.

D-JEPA introduces a decision-aligned latent world model: it first quantifies a decision-local prediction gap in which, among the few futures competing for execution, a candidate predicted closer to the goal can fail while an available alternative succeeds; it then uses executed outcomes to supervise a bounded, permutation-equivariant set operator that learns decision-relevant relations among candidate futures, and realizes the learned decision structure in JEPA-compatible future representations so aligned actions can be read out through native latent-distance planning; it reaches 87.89% success on PushT, a 15.04-point average gain on RoboTwin, a 17-point gain on physical robot tasks, and raises mean driving PDMS from 57.36 to 95.34.

D-JEPA introduces a decision-aligned latent world model: it first quantifies a decision-local prediction gap in which, among the few futures competing for execution, a candidate predicted closer to the goal can fail while an available alternative succeeds; it then uses executed outcomes to supervise a bounded, permutation-equivariant set operator that learns decision-relevant relations among candidate futures, and realizes the learned decision structure in JEPA-compatible future representations so aligned actions can be read out through native latent-distance planning; it reaches 87.89% success on PushT, a 15.04-point average gain on RoboTwin, a 17-point gain on physical robot tasks, and raises mean driving PDMS from 57.36 to 95.34.