# Activation Sparsity

The work introduces SpAx, which replaces the binary choice of whether to read a weight under activation sparsity with three options—fully skipping weights whose activations are closest to zero, reading compressed approximate weights for smaller-magnitude activations, and reading original weights for the largest-magnitude activations—thereby speeding up decoding when weights are offloaded: with weights offloaded to CPU memory, 3.86x average (up to 5.57x) for 16-bit weights and 2.06x average (up to 2.74x) for 4-bit weights; with weights offloaded to flash storage, 3.31x average (up to 4.81x) and 1.54x average (up to 2.03x), at a WikiText-2 perplexity increase of at most 10%.

The work introduces SpAx, which replaces the binary choice of whether to read a weight under activation sparsity with three options—fully skipping weights whose activations are closest to zero, reading compressed approximate weights for smaller-magnitude activations, and reading original weights for the largest-magnitude activations—thereby speeding up decoding when weights are offloaded: with weights offloaded to CPU memory, 3.86x average (up to 5.57x) for 16-bit weights and 2.06x average (up to 2.74x) for 4-bit weights; with weights offloaded to flash storage, 3.31x average (up to 4.81x) and 1.54x average (up to 2.03x), at a WikiText-2 perplexity increase of at most 10%.

TopK-Guided is a training-free activation sparsity method that combines bounded token-level sparsity adaptation with sensitivity-aware block-level budget allocation, consistently improving perplexity and downstream accuracy over TEAL and WINA across Llama-2 and Llama-3 models while preserving essentially the same sparsity-dependent projection compute as WINA, with the largest gains at high sparsity; ablations show the two components provide complementary improvements.

TopK-Guided is a training-free activation sparsity method that combines bounded token-level sparsity adaptation with sensitivity-aware block-level budget allocation, consistently improving perplexity and downstream accuracy over TEAL and WINA across Llama-2 and Llama-3 models while preserving essentially the same sparsity-dependent projection compute as WINA, with the largest gains at high sparsity; ablations show the two components provide complementary improvements.