Hear it, move now: Sonic-AMG cuts reaction latency from 10.9 s to about 193 ms

A body's reaction to sound can now be generated as the audio arrives, instead of waiting for a whole clip and then producing the whole motion at once. Earlier methods synthesize complete motion sequences inside predefined temporal windows, conditioned on fully observed acoustic and spatial inputs, so they cannot respond causally to an incoming audio stream. On the SAM dataset, Sonic-AMG reaches FID 6.505 and R@1 0.957 under the GT SSL protocol, better than MOSPA's 8.242 and 0.942; under the Online protocol it reaches FID 8.066 and R@1 0.943.

The model splits sound content and the source's location relative to the body into two conditioning paths, then lets a causal generator predict motion block by block. Acoustic content is encoded as 80-dimensional log-mel features through a shared causal temporal CNN, while the spatial side uses an 11-dimensional descriptor of direction, distance, their increments, inverse distance, direction change magnitude and looming, giving separate what and where tokens. Ablations show separate tokens beat fused tokens (FID 8.714 versus 11.445), and the learned log-mel features beat MOSPA's 2,272-dimensional audio features (FID 8.714 versus 12.759).

Training supervises the model on histories it generates itself, and feeds generated root poses back to update the spatial conditions. Two-forward training replaces selected ground-truth inputs with predictions from a preliminary teacher-forced pass, but those replacements are still predicted under ground-truth histories and do not recursively depend on earlier sampled outputs. Rehearsal training beats every two-forward schedule under both SSL protocols, with Online FID 8.714 versus 11.766 for the best two-forward variant; rollout-only training without full-sequence teacher-forced supervision rises to FID 17.036.

The report covers evaluation on the SAM dataset, where autoregressive models receive four initial ground-truth motion frames and the reaction style, and where source-location inputs are available to all methods in the main comparisons. The intended readers are researchers and engineers in motion generation, embodied interaction and real-time graphics. The authors propose future work to establish quantitative benchmarks for motion quality over longer audio streams with more diverse event transitions.

The main experiments assume known world-space source positions during inference, and the No-SSL variant is only an initial demonstration that spatial conditioning can be derived from binaural audio, with systematic evaluation of alternative spatial inference methods left as future work. Evaluation is limited to SAM, so generalization to broader acoustic environments and human reactions remains to be assessed, and evidence for long-horizon coherence and event transitions is primarily qualitative.