Diffusion models move temporal structure into the noising process: DSSD lowers forecast error on weather fields and robot videos
Synopsis
Diffusion models for spatiotemporal prediction usually add independent noise to each future frame and leave temporal dependence to the denoiser. DSSD instead builds the temporal operator of the corruption process from a learnable diagonal state-space model whose decay rates and frequencies train jointly with the denoiser, and uses the Cholesky factor of that model's stationary covariance so the operator stays causal, exactly invertible and unit-variance per frame. On SEVIR, Turbulence, BAIR and RoboNet it beats diffusion baselines sharing the same denoiser, with raw CRPS 3.975 versus 4.174 for the strongest SEVIR baseline, DPM.
Interpretation
A diffusion model for spatiotemporal prediction can put temporal correlation into the corruption process instead of relying only on the denoiser. DSSD mixes the noise of future frames through a learnable temporal operator, so neighboring frames carry decay and oscillation structure. The prior approach added temporally independent noise to each future frame and left temporal dependence entirely to the denoiser; DyDiff introduced correlation but only through a fixed exponential moving average whose decay follows a predefined schedule rather than the data, limiting it to one memory scale. On the SEVIR and Turbulence scientific-field datasets, DSSD reaches raw CRPS of 3.975 and 0.00756, against 4.174 for DPM and 0.01100 for DyDiff, the strongest baselines under the same denoiser; on BAIR it beats DPM and DyDiff on all eight reported metrics, and on RoboNet it leads six of eight.
The temporal operator is a set of damped oscillatory modes, each with a learnable decay rate and frequency, trained jointly with the denoiser. These modes come from a diagonal state-space model whose impulse response weights the current input and inputs several frames back, with weights given by the normalized modal kernel. Convolving over a finite window needs the input state before the window, and starting from zero leaves early frames with less noise variance than the schedule prescribes, for example half for the first frame; using the Cholesky factor of the stationary covariance keeps the operator causal, exactly invertible, unit-variance per frame, and the identity at the clean end.
Sampling undoes the current step's temporal mixing and reapplies the next step's, keeping the rest of the deterministic DDIM update. When the operator is the identity, the update reduces to the standard DDIM step, so temporal mixing sits on top of an existing sampler. On BAIR, DSSD's FVD is 766.0 against 1329.2 for DPM and 1417.4 for DyDiff; on RoboNet it is 430.2 against 659.2 and 770.6, while RMSE gains are modest.
Perspective
The demonstrated scope is fixed-window prediction on the SEVIR and Turbulence scientific-field datasets and the BAIR and RoboNet robot-video datasets, with all methods sharing one frozen VAE, one video U-Net or DiT denoiser and one 50-step DDIM sampling grid. The authors state they will release model weights, training and sampling code, and configurations.
Temporal mixing creates no information about the future at a single noise level, so any benefit can only come from the regression geometry a finite denoiser and sampler face, which the experiments rather than the theory establish. In finite precision an ill-conditioned learned operator can amplify errors, and the numerical checks cover only a fixed grid of dynamics levels and specified window lengths, not every parameter value.