ScanDreamer gives quadrotors an online map to imagine with: 96.3% success and 3.6% collisions in unseen simulated layouts

A quadrotor inspecting buildings and tunnels sees the world through a forward-facing depth camera with a limited field of view, and earlier world-model agents kept memory in a recurrent latent state without explicit geometric memory or awareness of where the camera points. At the same 0.8M environment-step budget, AirDreamer reaches 59.6% success and 28.8% collisions while ScanDreamer reaches 96.3% and 3.6%; training AirDreamer to 2.0M steps, 2.5 times the budget, does not close the gap. The comparison runs over 1560 test episodes in NVIDIA Isaac Sim covering unseen layouts and parameter ranges, with mean and standard deviation over three seeds and each seed paired with the same seed of the comparator.

The agent back-projects depth pixels with attitude into obstacle points and maintains a persistent occupancy grid in the take-off frame, so walls already seen stay on the map after the camera moves away. The map updates a 0.15 m grid with clamped log-odds and never clears occupied cells within an episode; a Dijkstra pass over the map recomputes geodesic cost every 10 steps or whenever occupied cells are added, treating unknown cells as traversable. On the topology-shift families that call for memory, a planner keeping only the last 3.2 s of observations solves at most 49% of 100 scenes per family, whereas the same planner with the full scanned map solves all of them.

Imagination is no longer purely latent: the scene is static, so an imagined action changes the pose but not the map, and progress, contact with mapped walls and arrival are computed directly from the frozen map at the imagined pose. Earlier world-model agents let learned heads for progress, wall contacts and arrival see neither the imagined pose nor mapped walls, so the actor could exploit their errors. The pose prior is fit on half of 291 episodes from AirDreamer's training replay and checked on the other half, where its median 15-step position error is 0.143 m against 0.363 m for an uncalibrated kinematic model.

A forward-cone penalty discourages horizontal motion outside the camera's field of view, and near-goal arrival handling keeps the critic from valuing the region just outside the arrival sphere above the arrival reward itself. Removing the forward-cone penalty raises collisions from 3.6% to 9.9% and drops the forward-motion share from 92.4% to 40.1%; removing arrival handling leaves success and collision without a consistent change but lengthens the final approach from 0.38 s to 1.39 s. Ablation variants are trained with the same setup as the full model, and metrics are reported as mean and standard deviation over three seeds.

All results come from NVIDIA Isaac Sim, where test arenas are 24 m by 24 m and obstacles are vertical boxes spanning the flight band, the setting the 2D map suits; the authors propose next steps of more diverse and realistic environments, a 3D map, noisy depth and visual-inertial odometry pose on a real quadrotor, and dynamic goals and obstacles.

Replacing only the Euclidean term of AirDreamer with the geodesic reward lowers success from 59.6% to 51.6%, and the authors hypothesize that this reward needs the map as an input; real buildings and subterranean sites also contain overhanging obstacles that a 2D map cannot express.