Streaming video-to-audio closes the alignment gap: LiveFoley needs only 0.08 s of frontend look-ahead and runs at 156 ms end-to-end
Synopsis
Real-time video-to-audio has been held back by two bottlenecks: visual frontends shrink their receptive fields to cut look-ahead, and causal diffusion backends denoise each patch in isolation, so both alignment and audio quality trail offline models. LiveFoley uses a trailing sliding window to turn off-the-shelf non-causal segment-level encoders into streaming modules without retraining, cutting frontend look-ahead to 0.08 s, and widens the backend denoising window toward history. On the VGGSound test set, LiveFoley-48k reaches a DeSync of 0.48, on par with offline MMAudio-L at 0.47, with 156 ms end-to-end latency on a single A800 GPU.
Interpretation
Streaming video-to-audio can now match offline models in temporal alignment at 0.08 s of frontend look-ahead while keeping the best audio quality among streaming systems. Earlier streaming systems had to trade one for the other: V-AURA uses segment-level encoders but looks ahead 0.64 s, SoundReactor cuts look-ahead to 0.08 s with per-frame features at a clear cost in alignment, and WorldSonus reaches a DeSync of 0.67 at 0.10 s. On the full VGGSound test set, LiveFoley-48k reaches a DeSync of 0.48, on par with offline MMAudio-L at 0.47 and ahead of all streaming baselines (0.67 to 1.03); LiveFoley-16k obtains the best Fréchet distance (9.56) and KL divergence (92.48) among streaming systems.
The frontend no longer buys low latency by shrinking its window; it anchors the tail of the segment window to the current frame and slides the window as new frames arrive. A chunked window must buffer a full window of frames before encoding, so look-ahead grows linearly with window length; shrinking the window from 4 tokens to 1 cuts look-ahead from 0.32 s to 0.08 s but narrows the temporal receptive field. On the VGGSound-Curated subset with V-AURA as the testbed, dropping look-ahead from 0.32 s to 0.08 s degrades the chunked window's DeSync from 0.67 to 0.88, whereas the trailing sliding window moves only from 0.60 to 0.61.
The backend widens the local diffusion head's denoising window toward history so neighboring audio patches see each other during denoising. Earlier streaming systems denoise each patch in isolation and see history only through the autoregressive hidden state, lacking cross-patch context, which makes training less stable and audio quality worse. In the ablation, History LocDiT lowers the Fréchet distance from 21.08 to 17.55 and raises the ImageBind score from 0.31 to 0.32 relative to Independent LocDiT, with no added backend look-ahead.
Perspective
The demonstrated setting is streaming inference on 25 fps video input on a single A800 GPU, with 156 ms end-to-end latency in the default configuration and 236 ms when Future-Warmup LocDiT adds 0.08 s of backend look-ahead; training used about 2,000 hours (745K clips), and the main evaluation is on the VGGSound test set.
Generation quality still trails offline models: LiveFoley-16k lags MMAudio-L in KL divergence (92.48 vs. 74.16), Fréchet distance (9.56 vs. 8.48), IS (11.30 vs. 11.62) and ImageBind (0.29 vs. 0.31), because a causal model must commit to each sound from the frames observed so far. In the listening test LiveFoley's audio-quality MOS-Q is 3.23, below MMAudio-L's 3.62, and the authors list closing this gap under the streaming constraint as an open direction.