Long video generation stops forgetting with time: Smart addresses a fixed-size slot memory by camera pose and still recalls old scenes beyond its training horizon

Long video generation can now remember scenes seen long ago under fixed memory and retrieve them by location when the camera returns, rather than by recency. Explicit geometric memories cache and reproject explicit 3D structure, so storage grows with the explored scene and recall depends on estimated 3D structure; recurrent memories keep a constant state, but retention is governed by temporal recurrence, so old observations fade with time. On long trajectories in Memory Maze and Minecraft, Smart maintains high-fidelity recall far beyond its training horizon while using constant memory and per-step cost; on Memory Maze, reconstruction quality degrades only gradually as trajectories lengthen, whereas all temporal-context baselines deteriorate more.

Both writing to and reading from the memory are decided by camera pose, and content never enters the addressing. Each region token's address comes from a convolutional encoding of the Plücker coordinates of that region's viewing rays, and write keys and read queries are per-head linear functions of that address alone, so a query matches a key only through the viewing geometry of the two regions. On writing, each token updates a single slot whose responsible pose region is fixed by a learned embedding, and unselected slots neither decay nor write; on reading, the target camera passes through the same address encoder to form a query, the memory returns a blend of slot contents, and the target camera re-poses that content into its own view together with a coarse draft and a per-cell confidence.

The generator treats the memory's coarse draft as a starting point and uses confidence to decide where to trust memory and where to fall back on its generative prior. The draft and its confidence mask are concatenated channelwise onto the current noisy latent, giving every token a spatially aligned memory estimate without adding tokens. The generator is a chunk-causal diffusion transformer whose each step runs a forward pass over only the current chunk and the clean context before it, so step cost depends on chunk length rather than on how long the video has run; generated chunks are written back into memory, so rollouts of arbitrary length run at constant memory.

The demonstrated setting is procedurally generated first-person mazes in Memory Maze, where revisits are enforced at low resolution, and open-terrain Minecraft trajectories at higher resolution with action-conditioned generation; the two instantiations share one memory interface but differ in backbone, autoencoder and resolution.

Fixed memory capacity bounds the spatial extent that can be retained: in sufficiently large environments different regions reuse the same slots, so previously stored content is overwritten and forgotten; the generator backbones used are relatively modest, and how the memory mechanism interacts with substantially larger video models and stronger generative priors has not been assessed.