# Behavioral compositionality of SAE representations

Sparse autoencoders split model activations into sparse, linearly decoded features, but combining those features does not make their downstream effects add. The work introduces SAE Additivity Residuals (SAR), which measure the behavioral variation left unexplained above a chosen interaction order; across 18 model and behavior settings, learned SAE coordinates show lower first-order residuals than sum-preserving random rotations of the same subspace, yet the coordinates are not fully additive, and pairwise terms improve held-out coalition ordering in all 18 settings. On 16 published steering directions in Llama-3.1 8B, pairwise modeling raises rank fidelity from 0.5874 to 0.7790 and cuts ordering error from 0.1444 to 0.0606.

Sparse autoencoders split model activations into sparse, linearly decoded features, but combining those features does not make their downstream effects add. The work introduces SAE Additivity Residuals (SAR), which measure the behavioral variation left unexplained above a chosen interaction order; across 18 model and behavior settings, learned SAE coordinates show lower first-order residuals than sum-preserving random rotations of the same subspace, yet the coordinates are not fully additive, and pairwise terms improve held-out coalition ordering in all 18 settings. On 16 published steering directions in Llama-3.1 8B, pairwise modeling raises rank fidelity from 0.5874 to 0.7790 and cuts ordering error from 0.1444 to 0.0606.