# Brain-Model Semantic Alignment

Similarity between multimodal large language model representations and human visual cortex shows that the two are alike but not which semantics they share. SAIL recasts voxel interpretation as a correspondence between each voxel and multiple sparse semantic units: it trains a unified cross-layer sparse autoencoder on several model layers, matches each voxel to several units, and compares the semantic similarity of their preferred image sets with optimal transport. Across eight subjects in the Natural Scenes Dataset, joint cross-layer training reduced unit response redundancy and raised semantic similarity over layer-wise training, in both whole cortex and selective regions of interest.

Similarity between multimodal large language model representations and human visual cortex shows that the two are alike but not which semantics they share. SAIL recasts voxel interpretation as a correspondence between each voxel and multiple sparse semantic units: it trains a unified cross-layer sparse autoencoder on several model layers, matches each voxel to several units, and compares the semantic similarity of their preferred image sets with optimal transport. Across eight subjects in the Natural Scenes Dataset, joint cross-layer training reduced unit response redundancy and raised semantic similarity over layer-wise training, in both whole cortex and selective regions of interest.