Konstantin Berestizshevsky

This work proposes Top-Theta (Top-j) Attention, a training-free inference-time method that replaces top-k search with static per-head calibrated thresholds to sparsify attention elements, and introduces Softmax Denominator Compensation (SDC) and V-Mean Compensation (VMC) to preserve accuracy, achieving 3–10× reduction in V-cache rows and up to 10× fewer attention elements on LLaMA2/LLaMA3 Q&A, code generation, and long-context summarization tasks with no more than 1% accuracy degradation.

This work proposes Top-Theta (Top-j) Attention, a training-free inference-time method that replaces top-k search with static per-head calibrated thresholds to sparsify attention elements, and introduces Softmax Denominator Compensation (SDC) and V-Mean Compensation (VMC) to preserve accuracy, achieving 3–10× reduction in V-cache rows and up to 10× fewer attention elements on LLaMA2/LLaMA3 Q&A, code generation, and long-context summarization tasks with no more than 1% accuracy degradation.