Skip to main content

# Video-to-Audio Generation

2 related articles

Subscribe to updates

arXiv Nov 4, 2026
Streaming video-to-audio closes the alignment gap: LiveFoley needs only 0.08 s of frontend look-ahead and runs at 156 ms end-to-end
Real-time video-to-audio has been held back by two bottlenecks: visual frontends shrink their receptive fields to cut look-ahead, and causal diffusion backends denoise each patch in isolation, so both alignment and audio quality trail offline models. LiveFoley uses a trailing sliding window to turn off-the-shelf non-causal segment-level encoders into streaming modules without retraining, cutting frontend look-ahead to 0.08 s, and widens the backend denoising window toward history. On the VGGSound test set, LiveFoley-48k reaches a DeSync of 0.48, on par with offline MMAudio-L at 0.47, with 156 ms end-to-end latency on a single A800 GPU.
Real-time video-to-audio has been held back by two bottlenecks: visual frontends shrink their receptive fields to cut look-ahead, and causal diffusion backends denoise each patch in isolation, so both alignment and audio quality trail offline models. LiveFoley uses a trailing sliding window to turn off-the-shelf non-causal segment-level encoders into streaming modules without retraining, cutting frontend look-ahead to 0.08 s, and widens the backend denoising window toward history. On the VGGSound test set, LiveFoley-48k reaches a DeSync of 0.48, on par with offline MMAudio-L at 0.47, with 156 ms end-to-end latency on a single A800 GPU.
Alibaba Group (China)Nanjing University Monash University Tampere University Tsinghua University
Yusheng Dai（First author）
Video-to-Audio Generation LiveFoley Trailing Sliding Window Autoregressive diffusion Flow-Matching Transformer
arXiv Oct 2, 2026
A survey unifies joint, cross-modal and joint-editing video-audio generation on one distribution and gives the first systematic taxonomy of joint audio-visual editing: nine categories, 28 edit types
Observing that video and audio are perceived together yet most generative models treat them in isolation, this survey casts joint generation, cross-modal generation (one modality from the other) and joint editing as three problems defined on a single distribution over audio-visual pairs, organizes the field around one question—how the output is kept coherent across modalities in time and semantics—proposes a taxonomy along five design axes (generation strategy, audio representation, video representation, alignment enforcement, pretraining reuse), and, to the authors' knowledge, provides the first systematic taxonomy of joint audio-visual editing, mapped as nine edit categories spanning 28 edit types, together with methods, datasets and metrics for each setting and a closing list of open pr
Observing that video and audio are perceived together yet most generative models treat them in isolation, this survey casts joint generation, cross-modal generation (one modality from the other) and joint editing as three problems defined on a single distribution over audio-visual pairs, organizes the field around one question—how the output is kept coherent across modalities in time and semantics—proposes a taxonomy along five design axes (generation strategy, audio representation, video representation, alignment enforcement, pretraining reuse), and, to the authors' knowledge, provides the first systematic taxonomy of joint audio-visual editing, mapped as nine edit categories spanning 28 edit types, together with methods, datasets and metrics for each setting and a closing list of open pr
University of Massachusetts Amherst Stanford University Dolby (United States)Texas A&M University University of Oregon Virginia Tech Vanderbilt University Arizona State University
Abhinav Sharma（First author）
Joint Audio-Visual Generation Joint Audio-Visual Editing Video-to-Audio Generation Cross-Modal Alignment

You’re all caught up