Sparse-autoencoder steering gets an information-theoretic design: DESSIRT beats a contrastive baseline on most question modes in Llama and Gemma

SAE steering-vector design shifts from picking features by contrastive statistics to minimizing divergence from a target dataset over feature firing-probability distributions. Earlier SAE steering followed contrastive activation addition, ranking and weighting features by dataset-wise activation-statistic differences, while feature interpretability and steering utility are not necessarily aligned. In the seven-mode question-generation task, DESSIRT-2 at layer 16 of Llama 3.1-8B-Instruct beat the SDCV baseline on all modes except M2, with M3 at 0.970 versus 0.900, M5 at 0.850 versus 0.490 and M6 at 0.895 versus 0.570, under a format-score threshold of 4.5.

The method writes each dataset as a vector of SAE feature firing probabilities, expands the divergence to second order for a virtual shift, and solves a closed form over a small feature support. Jensen–Shannon divergence serves as the target-alignment term, and the same divergence against a generic natural-language distribution serves as a stability budget for candidate features. The steering vector is decoded from feature space back to the residual stream by the SAE decoder; Llama vectors were built at budgets 10 and 30, and the Gemma-2-9B-IT experiment fixed 100 SAE features at layers 20 and 31, sweeping the coefficient from -180 to 300 in steps of 30.

Experiments cover two models, Llama 3.1-8B-Instruct and Gemma-2-9B-IT, with the main steering intervention at layer 16 in Llama and layers 20 and 31 in Gemma; evaluation uses a seven-mode question-generation task and a safety-related target–contrast response task, with text scored by an LLM-as-a-judge under custom rubrics.

Transfer to other intervention layers, models, SAE dimensions and sparsity settings remains to be studied, and the authors did not exhaustively search residual-stream extraction points, SAE training settings, feature-selection settings, the number of selected features or the budget limit. Text-level evaluation relies on an LLM-as-a-judge with custom rubrics, which may introduce model- and prompt-dependent bias, and the reference datasets are model-generated and automatically filtered, which may affect steering behavior.