# Activation Patching

The work decomposes an LLM's refusal behavior into a cross-layer pathway: harmful detection heads in early and middle layers pick out harmful tokens, safety neurons in mid-to-late layers carry and stabilize that signal in the residual stream, and refusal heads turn it into a refusal. Scaling the weights of these three component groups without changing architecture raised average safety under GCG attacks from 43.2% to 69.7% across six models, with a 1.7% average accuracy drop on four standard benchmarks.

The work decomposes an LLM's refusal behavior into a cross-layer pathway: harmful detection heads in early and middle layers pick out harmful tokens, safety neurons in mid-to-late layers carry and stabilize that signal in the residual stream, and refusal heads turn it into a refusal. Scaling the weights of these three component groups without changing architecture raised average safety under GCG attacks from 43.2% to 69.7% across six models, with a 1.7% average accuracy drop on four standard benchmarks.

Language models do not acquire in-context learning in one step. A controlled diagnostic separates copying the nearest demonstration from composing mappings spread across the prompt; across 17 models about a third prefer the copied answer and the rest the rule-consistent one, and in OLMo-2-7B dependence on the requested value overtakes the conflicting example hundreds of billions of tokens before accuracy rises above chance. Training on synthetic definition-and-query documents flips the preference, while exposure to definitions or suppressing copy-favoring heads does not.

Language models do not acquire in-context learning in one step. A controlled diagnostic separates copying the nearest demonstration from composing mappings spread across the prompt; across 17 models about a third prefer the copied answer and the rest the rule-consistent one, and in OLMo-2-7B dependence on the requested value overtakes the conflicting example hundreds of billions of tokens before accuracy rises above chance. Training on synthetic definition-and-query documents flips the preference, while exposure to definitions or suppressing copy-favoring heads does not.

Across 18 Base–Instruct model pairs, harmfulness is more decodable from attention updates while refusal evidence is written mainly by MLPs, and post-training strengthens that writing without improving recognition.

Across 18 Base–Instruct model pairs, harmfulness is more decodable from attention updates while refusal evidence is written mainly by MLPs, and post-training strengthens that writing without improving recognition.

Across five open-weight models, holding graph facts fixed and changing only the task instruction redistributes how the same attention heads influence later computation; in Gemma-2-9B and Qwen3-14B, editing one early internal activation vector switches which fact the model reads while the written instruction and weights stay fixed.

Across five open-weight models, holding graph facts fixed and changing only the task instruction redistributes how the same attention heads influence later computation; in Gemma-2-9B and Qwen3-14B, editing one early internal activation vector switches which fact the model reads while the written instruction and weights stay fixed.

The work proposes an empirical contract that evaluates LLM internal features at values observed on natural inputs through installation, removal, and downstream rescue: installation measures how far a feature suffices for a behavior, while removal and rescue measure how much the model uses it. Applied to published entity-recognition latents, dense known–unknown directions, and a subject–verb agreement feature set, sufficiency and use strengths separate sharply, showing that tracking a concept and steering a behavior do not by themselves show that the model uses a feature.

The work proposes an empirical contract that evaluates LLM internal features at values observed on natural inputs through installation, removal, and downstream rescue: installation measures how far a feature suffices for a behavior, while removal and rescue measure how much the model uses it. Applied to published entity-recognition latents, dense known–unknown directions, and a subject–verb agreement feature set, sufficiency and use strengths separate sharply, showing that tracking a concept and steering a behavior do not by themselves show that the model uses a feature.

The work introduces the Unlearning Depth Score (UDS), which first uses a retain-model baseline to identify layers encoding target knowledge and then measures via activation patching how much of that knowledge the unlearned model has erased on a 0-1 scale; in a meta-evaluation across 20 metrics on 150 unlearned models spanning 8 methods, UDS is reported to achieve the highest faithfulness and robustness, and case studies show it uncovers residual knowledge obscured from observational metrics by representational shifts, with erasure depth varying across prompt types.

The work introduces the Unlearning Depth Score (UDS), which first uses a retain-model baseline to identify layers encoding target knowledge and then measures via activation patching how much of that knowledge the unlearned model has erased on a 0-1 scale; in a meta-evaluation across 20 metrics on 150 unlearned models spanning 8 methods, UDS is reported to achieve the highest faithfulness and robustness, and case studies show it uncovers residual knowledge obscured from observational metrics by representational shifts, with erasure depth varying across prompt types.

The work introduces Patch-to-Prune (P2P), a training-free framework that converts activation patching from a diagnostic tool into an inference-time computation bypass: validation-guided forward and backward layer sweeps identify decoder regions whose visual-token projection outputs can be replaced by fixed neutral proxy activation vectors within a user-specified accuracy tolerance; evaluated on four VLMs from the Qwen2.5-VL and LLaVA families across seven multi-modal benchmarks with mutually disjoint calibration, validation, and test partitions, P2P at a 3% tolerance retains around 94% of dense accuracy while reducing FLOPs by 55%, and its layer-wise analysis suggests visual processing is non-uniformly distributed across decoder depth.

The work introduces Patch-to-Prune (P2P), a training-free framework that converts activation patching from a diagnostic tool into an inference-time computation bypass: validation-guided forward and backward layer sweeps identify decoder regions whose visual-token projection outputs can be replaced by fixed neutral proxy activation vectors within a user-specified accuracy tolerance; evaluated on four VLMs from the Qwen2.5-VL and LLaVA families across seven multi-modal benchmarks with mutually disjoint calibration, validation, and test partitions, P2P at a 3% tolerance retains around 94% of dense accuracy while reducing FLOPs by 55%, and its layer-wise analysis suggests visual processing is non-uniformly distributed across decoder depth.

Using activation patching across twenty-five models spanning eight LLM families, the work identifies an early-layer (L0) attention routing circuit shared by VQ-tokenized vision-language models, proposes a three-gate diagnostic that isolates ten models carrying it and rejects the other fifteen, shows via a single-variable architectural swap (LLaVA-1.6 CLIP+MLP to VQ+Linear) that vector quantization is the source of the pathological signal, and finds that only L0 ablation reduces object hallucination in open-ended generation (CHAIR_i down 31% relative) while tuned DoLA and VCD do not.

Using activation patching across twenty-five models spanning eight LLM families, the work identifies an early-layer (L0) attention routing circuit shared by VQ-tokenized vision-language models, proposes a three-gate diagnostic that isolates ten models carrying it and rejects the other fifteen, shows via a single-variable architectural swap (LLaVA-1.6 CLIP+MLP to VQ+Linear) that vector quantization is the source of the pathological signal, and finds that only L0 ablation reduces object hallucination in open-ended generation (CHAIR_i down 31% relative) while tuned DoLA and VCD do not.