Safety alignment no longer rests on a few thousand neurons: ABS-Align prunes them during finetuning and cuts some attack success by about 90%

The fragility of safety alignment is treated as a trainable structural problem: the finetuned model no longer keeps refusal behavior concentrated in the original safety-critical neurons, so pruning those neurons or ablating the refusal direction is less effective. Prior work showed that pruning safety-critical neurons, which make up only a tiny fraction of the model, or ablating a single refusal direction from intermediate activations can largely remove safety alignment while barely affecting general capabilities; existing input/output filtering and finetuning defenses did not change this sparsity. On Gemma-2-2B-IT, the cumulative StrongREJECT score of TwinBreak over five pruning iterations falls from 0.71 without defense to 0.23; Llama-3-8B-Instruct falls from 0.81 to 0.11, Llama-3.2-3B-Instruct from 0.74 to 0.33, and Qwen-3-4B from 0.79 to 0.13.

The procedure locates first and removes gradually: the model loses its original safety neurons bit by bit during finetuning and can only maintain refusal behavior through the remaining neurons and representations. The method scores neurons with TwinBreak from activation differences between malicious and benign twin prompts, ranks them by score, and prunes the ones most important to safety alignment first. Training uses 1,000 preference samples randomly drawn from PKU-SafeRLHF plus 850 Alpaca samples and 150 safe XSTest samples, finetuned with the direct preference optimization loss, with the first and last Transformer blocks frozen and the next batch of safety neurons pruned every 200 steps.

The demonstrated scope is four open-weight models, Gemma-2-2B-IT, Llama-3-8B-Instruct, Llama-3.2-3B-Instruct and Qwen-3-4B; the threat model is an attacker with white-box access who can prune neurons or intervene on activations, and adversarial finetuning attacks are out of scope. Results are scored with the StrongREJECT and SORRY-Bench evaluators, and training uses only 1,000 preference samples plus a small set of capability-retention samples.

Safety-neuron detection relies on heuristic scoring, so it may miss some safety neurons or mislabel unrelated ones, and the absolute density of safety alignment after finetuning is not directly quantified; the authors use "denser" to mean reduced dependence on the originally identified neurons, not a measured increase in density. Under an adaptive attack that raises TwinBreak's pruning budget from 5 to 15 iterations, Qwen-3-4B's attack success rate rises from 0.13 to 0.29, still below the undefended 0.79 but showing that a larger budget can partly restore attack effectiveness.