# Sink-aware pruning

The work identifies the BOS sink phenomenon as a key mechanism driving layer-wise redundancy sensitivity in large language models, showing that attention heads with high BOS sink scores, especially in deeper layers, contribute little to predictive performance and effectively serve as dumping grounds for superfluous attention weights, and introduces a simple pruning strategy that removes high-BOS sink heads; experiments on Gemma-3, Llama-3.1, and Qwen3 show this approach identifies redundant transformer components more reliably than weight- and activation-based criteria in terms of downstream task retention, remaining close to dense baselines at low-to-moderate pruning ratios, and that high-scoring sink heads sustain their focus on BOS as context length grows.

The work identifies the BOS sink phenomenon as a key mechanism driving layer-wise redundancy sensitivity in large language models, showing that attention heads with high BOS sink scores, especially in deeper layers, contribute little to predictive performance and effectively serve as dumping grounds for superfluous attention weights, and introduces a simple pruning strategy that removes high-BOS sink heads; experiments on Gemma-3, Llama-3.1, and Qwen3 show this approach identifies redundant transformer components more reliably than weight- and activation-based criteria in terms of downstream task retention, remaining close to dense baselines at low-to-moderate pruning ratios, and that high-scoring sink heads sustain their focus on BOS as context length grows.