World models stop growing with video length: L3G-World keeps a latent 3D Gaussian memory and runs at 9.02 FPS on one H100
Synopsis
Interactive world models have been limited by memory that grows with stream length: temporal context runs out of GPU memory, and explicit 3D memories slow down as redundant primitives accumulate frame by frame. L3G-World replaces that with a compact latent 3D Gaussian memory that spawns Gaussians only in newly observed regions, refines existing ones with residual updates, and stores a VAE latent feature in each Gaussian so the memory renders directly into the diffusion transformer's latent space without re-encoding. On 481-frame loop-closure trajectories from DL3DV and RealEstate10K, full L3G-World reaches PSNR of 16.00 and 18.36, and with few-step distillation it generates at 9.02 FPS on a single H100.
Interpretation
A world model's scene memory now grows with explored spatial extent rather than with the number of generated frames. Temporal-context methods kept history frames or self-attention KV caches as memory, and explicit 3D-memory methods accumulated point-cloud or Gaussian primitives per frame; both became more expensive as the generated stream lengthened. On DL3DV, L3G-World stores 33K primitives in 11.88 MiB, while the Gaussian baseline NeoVerse stores 8,766K primitives in 501.57 MiB; on RealEstate10K the pair is 25K versus 10,437K and 8.91 MiB versus 597.21 MiB.
The memory updater spawns new Gaussians only in newly observed regions and applies residual corrections to Gaussians in explored regions, so the memory is a continually revised scene representation rather than an accumulating buffer. Per-frame feed-forward reconstruction piled redundant primitives onto the same spatial regions and left stored primitives unupdated, so geometric errors persisted. In the DL3DV novel-view-synthesis ablation, removing adaptive spawning raised the Gaussian count from 41K to 75K and lowered PSNR from 16.52 to 16.26; removing the refinement stream lowered PSNR to 15.52.
Each Gaussian also carries a VAE latent feature, so the memory renders straight into the video diffusion transformer's latent space and skips rendering RGB followed by VAE encoding. Earlier explicit 3D memories stored only RGB appearance, so conditioning a latent video generator required re-encoding rendered output, which precluded real-time generation. On DL3DV, L3G-World renders 1,086.58 FPS in VAE space, against 3.11 FPS for the most efficient Gaussian baseline, NeoVerse; in the ablation, removing VAE lifting drops VAE rendering throughput to 2.97 FPS.
With few-step distillation of the memory-conditioned video generator, world generation runs in real time at 9.02 FPS on one H100. None of the explicit 3D-memory methods in the comparison table reached real time, with the highest total FPS being NeoVerse at 1.99. The comparison table records L3G-World at 9.02 total FPS and 7.4 GiB peak GPU memory and marks it real-time; the other listed methods are not marked real-time.
Perspective
The demonstrated setting is single-image input with generation along given camera trajectories on DL3DV and RealEstate10K; efficiency is measured on a single A100, and the real-time figure comes from the few-step distilled variant on a single H100.
The memory is designed mainly for static scenes and does not explicitly model independently moving objects; it can still grow under extremely large-scale exploration; and errors introduced during long-horizon autoregressive generation may still propagate into later memory updates.