Audio diffusion transformer takes fMRI directly: skipping intermediate embeddings lifts both acoustic and semantic reconstruction scores
Synopsis
MAAS replaces the text-embedding conditioning of a pretrained audio diffusion transformer with region-of-interest fMRI responses, removing the decoder that previously predicted an external stimulus representation such as CLAP or YamNet. On a 7T fMRI dataset it improves acoustic and semantic metrics, including CLAP similarity, pairwise identification accuracy and Mel-SSIM, over baselines such as two-stage fMRI-to-latent conditioning. The framework also uses a fidelity metric grounded in a computational model of auditory cortex to quantify how individual auditory regions contribute to the reconstructed audio.
Interpretation
Audio reconstruction no longer has to translate brain activity into an intermediate embedding such as CLAP or YamNet before handing it to a generative model; a pretrained audio diffusion transformer can be conditioned directly on fMRI responses organized by cortical region. Earlier brain-to-audio pipelines generally used a two-stage strategy, first mapping neural activity onto intermediate deep-network embeddings and then using those to guide a generative model; direct conditioning had been explored in vision but not yet extended to audio diffusion models. On the 7T fMRI dataset, MAAS scored higher than baselines including two-stage fMRI-to-latent conditioning on CLAP similarity, pairwise identification accuracy and Mel-SSIM, while also giving the lowest log-mel distance.
Brain signals enter the diffusion transformer through a dedicated cross-attention pathway parallel to the text stream, and each region keeps its own projection, so the influence of different cortical areas on the generated sound can be separated. Models such as Stable Audio natively accept only T5 text embeddings, apply rotary positional embeddings only to latent self-attention queries, and use fewer key-value heads than query heads, so adapters built for UNet architectures do not transfer directly. The experiments use a 7T dataset of five participants and 288 one-second natural sounds, evaluated with four-fold cross-validation on 72 held-out sounds per fold, alongside controls that shuffle brain-stimulus pairings and retrieve stimuli from neural signals alone.
Perspective
The demonstrated scope is a 7T fMRI dataset of five participants and 288 one-second natural sounds, evaluated with four-fold cross-validation on 72 held-out sounds per fold; the authors note that the dataset size constrains how far the reconstruction results generalize.
The BOLD signal integrates neural activity over several seconds, so MAAS receives a temporally blurred snapshot of auditory cortical activity; the authors state that the relationship between attention routing and the ablation outcomes warrants further investigation.