SteerScope benchmark: none of 23 steering methods beats the Prompt Steering baseline on the efficacy–side-effect trade-off
Synopsis
The authors introduce SteerScope, a two-axis, 15-metric evaluation suite for LLM activation steering, and benchmark 23 methods spanning 4 families (including prompting, LoRA, and SFT baselines) on Gemma-2-2B-it and Gemma-2-9B-it under matched models, tasks, and protocols; they find that no activation steering method achieves higher target efficacy without greater composite side effects, so none surpasses the Prompt Steering baseline on the efficacy–side-effect trade-off, and that efficacy and side effects rise together with intervention strength.
Figure 1: Overview of the SteerScope evaluation suite and benchmark coverage. Left: our hierarchical evaluation taxonomy. Right: coverage matrix of existing steering benchmarks, where rows denote taxonomy nodes and columns denote evaluation works (last column: SteerScope). A ✓ indicates a corresponding metric, ✗ indicates none, and △ \triangle indicates that the dimension is considered indirectly but not measured independently.
arXivInterpretation
SteerScope organizes 15 metrics into two complementary axes: Steering Outcomes measures target efficacy (Concept Expression) plus side effects on language quality, task capabilities, and safety and reliability; Method Properties measures generalization under cross-lingual prompt shifts and dependence on training data. Existing benchmarks cover these dimensions only in fragments: AxBench limits off-target evaluation to instruction following and fluency, SteeringSafety centers on safety entanglement, CLaS-Bench specializes in cross-lingual control, and FaithSteer-BENCH evaluates at a single calibrated operating point without covering language-quality or safety side effects. SteerScope brings these dimensions under one matched protocol. The 15 metrics comprise eight evaluators and eleven top-level side-effect metrics; all methods use exactly the same evaluation examples for a given concept, while different concepts use different deterministic subsets. Inexpensive benchmarks (MMLU, BBQ, TruthfulQA) cover all 500 concepts, while expensive ones (SuperGLUE, MATH, IFEval, JailbreakBench) use a 50-concept subset shared across methods and factors.
Across both model scales, no evaluated activation steering method surpasses the Prompt Steering baseline on the efficacy–side-effect trade-off: none achieves higher target efficacy without greater composite side effects. Some activation steering methods do outperform Prompt Steering on existing benchmarks, for example A-PSR on AxBench's Overall and Concept Expression metrics; that advantage does not hold once a broader range of side effects is considered. On Gemma-2-9B-it, A-PSR improves Concept Expression by 1.30 points at its Overall-selected method-level factor while incurring an average normalized degradation of 0.12; on Gemma-2-2B-it the corresponding figures are 1.20 points and 0.11. Under symmetric Dirichlet resampling of metric weights, Prompt Steering is almost never dominated under near-uniform weights (undominated in 99.83% and 99.92% of draws), and the conclusion remains consistent though less robust under highly sparse weightings.
Steering efficacy and side effects generally increase together as intervention strength grows, and low-strength interventions sometimes improve off-target metrics; those gains disappear or reverse when the steering direction is reversed, pointing to direction-dependent conceptual entanglement rather than a generic effect of weak perturbations. This supplies signed-factor evidence for the concept-entanglement account of side effects rather than a purely correlational description. MMLU was evaluated on Gemma-2-2B-it at layer 20 using signed normalized factors; gains produced by some positive interventions disappeared or became degradations when their steering directions were reversed.
Under cross-lingual prompt distribution shifts, target efficacy is often preserved while side effects tend to become more pronounced; methods also differ substantially in sample efficiency. Generalization and data dependence are measured as method properties rather than inferred from absolute scores at a single operating point. Among the 13 methods shown in Table 1, Instruction Degradation is larger under OOD prompts for 11 and Fluency Degradation is larger for 9. On sample efficiency, HyperSteer and DiffMean retain a relatively large fraction of their full-data effect with only a few examples, whereas A-PSR, FLAS, LoRA, and ReFT-r1 require more data to approach full-data performance.
Perspective
The suite is aimed at researchers and engineers who need inference-time control of LLM behavior and care about the cost of side effects; it applies to instruction-tuned Gemma-2-2B-it and Gemma-2-9B-it, the CONCEPT500 concept set, and single-site interventions on the residual stream at layer 20. Because it reports strength-dependent efficacy–side-effect curves, it supports method selection and operating-point choice: activation steering remains valuable when a high-efficacy operating point is the goal, while the Prompt Steering baseline is the current reference for overall balance. Cross-lingual generalization is evaluated only on text-type concepts, since cross-lingual prompt shifts do not apply to code or math concepts.
A careful reader would still watch several things: the composite side-effect score weights the eleven metrics equally, and although the authors test robustness with Dirichlet weight resampling, the conclusion becomes less robust under highly sparse metric preferences; the method-level factor is selected on the full ID evaluation set, which the authors note may introduce some optimism in operating-point calibration; retention and sample efficiency are ratios and become unstable when the denominator is close to zero, so they are reported only for subsets meeting a threshold; the confidence intervals capture concept-sampling uncertainty only, not training-seed or additional generation and judging uncertainty; and SFT is trained and evaluated on 20 concepts for compute reasons.