Self-driving planners no longer learn only from logged routes: simulated futures for candidate paths lift NAVSIM v1 PDMS from 86.2 to 91.3

During post-training, the planner samples candidate routes beyond the recorded trajectory, and those candidates previously had no matching future video to learn from, leaving driving reward as the only signal. Driving logs store future video only for the trajectory that was actually recorded, so when a candidate turns while the recorded trajectory continues straight, the logged video does not show the visual consequences of that action. Shuffling rendered targets across trajectory conditions within the same scene drops PDMS from 91.3 to 83.1, indicating the gain depends on correct trajectory-video pairing rather than on added video supervision alone.

The method first renders future videos for high-reward candidates with a simulator, then uses a flow-matching loss so video prediction and trajectory planning update the same shared representation. Because the rendering budget is limited, candidates are first ranked by trajectory reward, and among the rendered ones video supervision is prioritized for futures with higher prediction error. On NAVSIM v1, letting the video loss update only the video network gives 89.1 PDMS, while also updating the shared encoder gives 91.3; doubling the rendering budget from four candidates to eight gives 91.2, essentially level with 91.3.

The framework is validated on the Epona backbone, with post-training on NavTrain samples that have available World Engine scene reconstructions, rendered by a 3D Gaussian Splatting-based simulator. Evaluation covers the NavTest splits of NAVSIM v1 and v2 and closed-loop driving on the same 345 pre-challenge HUGSIM scenarios, using PDMS, EPDMS, route completion and HD-Score. At inference the model predicts trajectories directly from observation history, without simulator calls, future-video generation, or video-based reranking.

Occlusions and limited pose coverage can degrade renderings away from logged paths, while renderer metrics assess only logged viewpoints, and rendering artifacts can still appear even with safety and pose-support checks. A high prediction error can stem from either model limitations or poor-quality simulation targets. Observation histories remain log-based and rendered targets can differ from real images, and teacher-forced target selection cannot capture autoregressive error accumulation. The authors leave robustness evaluation under shifted observation histories and reactive traffic for future work, and note the framework is validated only on the Epona backbone, with generalization to alternative world action model architectures also left for future work.