AudioSphere learns spatial audio from multichannel self-supervision: a frozen backbone localizes real-world sound events for the first time
Synopsis
AudioSphere extends self-supervised audio pretraining from single-channel to multichannel: it takes four-channel Ambisonics recordings and trains a masked autoencoder to reconstruct both spectrograms and intensity vectors, so a frozen backbone with a shallow probe performs sound event detection and localization in real scenes. Prior single-channel models were blind to spatial structure, while spatial models relied on clip-level supervision. It reaches an aggregate HEAR score of 83.2 and also supports frame-level localization.
Interpretation
Self-supervised audio representations no longer have to be learned from single-channel signals alone: a frozen multichannel pretrained backbone can output frame-level directional information for detecting and localizing sound events in real recordings. Earlier self-supervised audio embedding models were trained almost exclusively on single-channel corpora such as AudioSet and LibriSpeech and were effectively blind to spatial structure, while spatial models such as BAT and ELSA captured directional cues only through clip-level supervised labels or contrastive spatial metadata, with objectives not designed for frame-resolved representations. On TUT-2018 Real, the frozen backbone with a shallow MLP probe reaches F-scores of 82.1, 74.7 and 71.1 at overlaps 1, 2 and 3, with error rates of 0.29, 0.38 and 0.42, all better than the fully supervised SELDNet, MSEDnet and SEDnet baselines in the table; on TAU-2019 it reaches error rate 0.28 and F 84.6, against 0.34 and 79.9 for the in-domain supervised SELDNet.
The model learns acoustic content and directional cues inside one reconstruction objective, so the same embeddings serve both clip-level classification and frame-level detection. The method is a multichannel masked autoencoder whose inputs are log-mel spectrograms of four-channel first-order Ambisonics plus intensity vectors, directional vectors computed from inter-channel phase and magnitude relations. Training uses tube masking, which selects 80% of time-frequency locations at random and masks them across all channels at once, forcing the encoder to infer directional channels from surrounding spatial context; pretraining data mixes 70,000 Ambisonic room impulse responses simulated from Matterport3D scenes with SoundSpaces 2.0 and AudioSet clips online.
Perspective
The demonstrated scope is sound event detection and direction-of-arrival estimation on first-order Ambisonics input, evaluated on TUT-2018 Real, TAU-2019, TAU-2020, TAU-2021, STARSS23 and the planar-array RSL2019-dev; distance estimation, source separation, dereverberation and room-acoustic inference are not evaluated.
Pretraining simulates static sources only, so the frozen representations lack the temporal-continuity priors needed to track moving sources and require a temporal probe head or fine-tuning; Ambisonics encoding quality is also bounded by the spatial sampling capacity of the physical array, leaving a residual localization gap for tetrahedral arrays relative to the 32-channel Eigenmike.