World Observer keeps out-of-view objects evolving via panoramic observers, lifting OOV-D from 0.358 to 0.531

It proposes joint actor-observer world modeling: an actor renders the agent-centric view while one or more panoramic observers maintain world state beyond it, both generated jointly by a single pretrained video diffusion transformer in the latent space of a 3D VAE and sharing one world state. Prior video world models are actor-centric, so once an object leaves the view its evolution can only be inferred from internal memory, generative priors, or explicit state extrapolation; here out-of-view regions are directly represented by a continuously generated panoramic stream rather than inferred from previously observed information. The paper provides the method derivation and ablations: removing observer generation yields lower OOV-D and OOV-D, which the authors interpret as the actor tending to bring previously unseen objects back with an inconsistent state, an impostor tendency, whereas joint generation achieves the highest OOV-D with a stable OOV-F.

It grounds geometry by warping from a shared panoramic source and introduces an Observer Sink of high-resolution perspective references to restore fine appearance when a region re-enters the actor view. A panorama carries projection distortion that loses fine appearance and weakens actor-observer correspondence; the Observer Sink crops four perspective views 90 degrees apart from the initial panorama and appends them as fixed latents that generated tokens can attend to. Ablation shows that without the Observer Sink OOV-D drops and 3D adherence, FID, and FVD degrade; the authors also report that on a trajectory rotating through a full 360 degrees, fine appearance degrades as rotation grows when this reference is absent.

Observers are decoupled from the actor, so they can be placed freely, extended to multiple locations, and driven by control signals to steer out-of-view evolution. An observer that follows the actor is constrained by the actor trajectory and cannot independently monitor a region of interest; decoupling allows fixed observers watching regions the actor cannot see, or multiple observers covering occluded and spatially separated regions. The multi-observer model starts from the single-observer checkpoint and reports OOV-F rising from 0.580 to 0.722 while both OOV-D metrics remain competitive; synthetic CARLA data supplies the time-synchronized multi-observer configurations that real captures rarely contain.

It introduces world-space out-of-view dynamics metrics OOV-F, OOV-D, and OOV-D, plus a benchmark spanning real and synthetic scenes. Existing benchmarks largely rely on VLM judgments of whether a reappearance looks natural, but the authors note that when both camera and objects move, VLMs cannot read an object's true 3D motion from frames and misjudge frozen or impostor states as natural; this work lifts objects into 3D with a segmentation model and a metric-scale depth estimator and measures displacement explicitly. The paper shows two VLM misjudgments: in the frozen case the VLM answers yes although the object has actually stopped, and in the impostor case it also answers yes although the reappearing motion no longer follows the object's own prior trajectory; OOV-D and OOV-D are respectively sensitive to these two failures.

The work targets world-model settings that must continuously maintain the state of unobserved regions, such as embodied navigation, interactive simulation, and long-horizon planning, where relevant regions should remain represented even when the actor is not currently looking at them. The method currently assumes panoramic observations as conditioning inputs, so it applies where such views are available; the authors propose first outpainting a perspective observation into a panorama so the same framework can operate with perspective-only inputs, and multi-observer settings could likewise construct panoramic conditions from perspective observations at different locations. Because the observer budget is limited, observing the entire world remains difficult, but flexible placement lets that budget be allocated to regions of interest so their dynamics are preserved independently of where the actor moves. Evaluation is conducted within a single chunk, where the full sequence fits in context and no memory retrieval is needed, so the results speak to whether the model genuinely maintains out-of-view dynamics rather than to long-context or retrieval ability.

The paper assumes panoramic observations as conditioning inputs; whether outpainting perspective inputs into panoramas preserves comparable behavior remains open. The observer budget is limited, so covering the entire world stays difficult, and how multi-observer configurations scale in more complex scenes is still to be observed. Evaluation runs within a single chunk, and although long-horizon autoregressive behavior is shown qualitatively, the quantitative conclusions come mainly from the single-chunk setting. The metrics depend on a segmentation model and a metric-scale depth estimator, so how their errors propagate into OOV scores deserves further attention. In addition, this evidence bundle is a full-text parse, but some figures and tables appear as prose descriptions, so exact numerical details should still be checked against the original tables.