Driving planners get the missing future frames: rendering along candidate trajectories lifts NAVSIM v1 from 86.2 to 91.3
Synopsis
Driving world action models explore candidate trajectories absent from logs during reinforcement-learning post-training, but logs store future video only for recorded trajectories, so those candidates lack matched visual supervision. SimVTA renders future video along candidate paths with a simulator that reconstructs real scenes via 3D Gaussian Splatting, giving prediction-independent targets, then selects high-reward, high-error futures by trajectory reward and video prediction error for flow-matching supervision while GRPO optimizes planning. On NAVSIM v1 it raises PDMS from 86.2 to 91.3, reaches 88.2 EPDMS on NAVSIM v2, and averages 38.8 HD-Score on HUGSIM.
Interpretation
During post-training the planner samples candidate trajectories that do not exist in the logs, and those candidates previously had no matching future-video supervision. Driving logs keep future video only for recorded trajectories, so when a candidate turns while the recorded trajectory goes straight, the logged video does not show that action's visual consequences; reusing it creates wrong trajectory-video pairs, while dropping video training leaves candidates without visual supervision. On NAVSIM v1, adding video supervision lifts PDMS from 89.1 with trajectory GRPO and BC alone to 91.3, while letting the video loss update only the video network and not the shared encoder stays at 89.1.
The method first renders future video for high-reward candidates with a simulator, then keeps the high-reward, high-error futures for video supervision. Because the rendering budget is limited, candidates are ranked by trajectory reward for rendering, and among rendered candidates only those whose prediction error is at or above the scene mean, the Q2 subset, are supervised. Under the same four-candidate rendering budget, Q2 flow-matching supervision reaches 91.3 PDMS, expanding supervision to Q1+Q2 gives 91.0, video GRPO over all valid rendered candidates gives 91.0, and shuffling rendered targets within a scene drops PDMS to 83.1.
Video supervision back-propagates into the shared representation used for planning, not only into the video network. Previously video prediction and trajectory planning could be trained separately, with the video loss leaving the planning representation unchanged. PDMS is 89.1 when the video loss updates only the video network and 91.3 when it also updates the shared encoder, and that advantage holds across three random seeds.
Perspective
The post-training is validated on NAVSIM v1, NAVSIM v2 and HUGSIM, and at inference the planner predicts trajectories directly without simulator calls or future-video generation. Training renders future video along candidate paths with the 3D Gaussian Splatting simulator in World Engine, starting from an Epona checkpoint. The authors list generalization to alternative world action model architectures and robustness evaluation under shifted observation histories and reactive traffic as future work.
Rendering quality degrades under occlusions and limited pose coverage away from logged paths, while renderer metrics assess only logged viewpoints; a high prediction error can come from model limitations or from poor-quality simulation targets. Observation histories remain log-based, rendered targets differ from real images, and teacher-forced selection cannot capture autoregressive error accumulation. On NAVSIM v2, the aggregate EPDMS gain coexists with lower traffic-light compliance, 99.7 to 99.0, and extended comfort, 67.6 to 60.0.