Brain voxels get more than one semantic unit: SAIL merges cross-layer sparse features for lower redundancy and higher alignment
Synopsis
Similarity between multimodal large language model representations and human visual cortex shows that the two are alike but not which semantics they share. SAIL recasts voxel interpretation as a correspondence between each voxel and multiple sparse semantic units: it trains a unified cross-layer sparse autoencoder on several model layers, matches each voxel to several units, and compares the semantic similarity of their preferred image sets with optimal transport. Across eight subjects in the Natural Scenes Dataset, joint cross-layer training reduced unit response redundancy and raised semantic similarity over layer-wise training, in both whole cortex and selective regions of interest.
Interpretation
The correspondence between cortical voxels and a multimodal large language model can now be attached to specific semantic units rather than only to overall representational similarity. The prior approach trained sparse autoencoders independently at each layer and matched each voxel to a single best unit, yielding one dominant preference. Across eight subjects in the Natural Scenes Dataset, the jointly trained SAIL beat the layer-wise SAEs-BrainMap on both unit response redundancy and semantic similarity, evaluated over whole cortex and selective regions of interest.
The method concatenates representations from several model layers into one sparse dictionary, so units from different layers compete in a shared space instead of each layer learning its own. The instrument is a cross-layer sparse autoencoder, an autoencoder that compresses high-dimensional activations into a few nonnegative sparse coefficients, which are the unit activations. Each layer is reduced by PCA and standardized before concatenation; matching takes the top K units by cosine similarity, and semantic agreement is scored on held-out images with SEED image similarity plus discrete optimal transport.
The matched units can in turn predict category responses in an independent functional localizer experiment. Prediction holds the voxel-unit correspondences fixed and weights unit activations by normalized matching scores. Across eight subjects and eight localizer categories, mean cosine similarity between predicted and measured responses ranged from 0.660 to 0.812, lowest for the number category and highest for the child category.
Perspective
The findings rest on static images from the Natural Scenes Dataset and eight subjects, covering four multimodal large language models; the semantic labels and preferred images describe semantic preferences within that stimulus set, and the cortical distribution of action-related semantics is an exploratory localization rather than an independent demonstration of action selectivity.
The authors note that preferred images and labels characterize semantic preferences within the evaluated stimulus set but cannot fully separate an individual concept from co-occurring visual content, which would need controlled stimulus comparisons; the analysis also covers only static images, leaving how brain-model correspondence evolves over time under dynamic stimuli unexamined.