Yusheng Dai

Real-time video-to-audio has been held back by two bottlenecks: visual frontends shrink their receptive fields to cut look-ahead, and causal diffusion backends denoise each patch in isolation, so both alignment and audio quality trail offline models. LiveFoley uses a trailing sliding window to turn off-the-shelf non-causal segment-level encoders into streaming modules without retraining, cutting frontend look-ahead to 0.08 s, and widens the backend denoising window toward history. On the VGGSound test set, LiveFoley-48k reaches a DeSync of 0.48, on par with offline MMAudio-L at 0.47, with 156 ms end-to-end latency on a single A800 GPU.

Real-time video-to-audio has been held back by two bottlenecks: visual frontends shrink their receptive fields to cut look-ahead, and causal diffusion backends denoise each patch in isolation, so both alignment and audio quality trail offline models. LiveFoley uses a trailing sliding window to turn off-the-shelf non-causal segment-level encoders into streaming modules without retraining, cutting frontend look-ahead to 0.08 s, and widens the backend denoising window toward history. On the VGGSound test set, LiveFoley-48k reaches a DeSync of 0.48, on par with offline MMAudio-L at 0.47, with 156 ms end-to-end latency on a single A800 GPU.