Bohao Tang

Researchers from Shanghai Jiao Tong University and ByteDance Seed propose PISA, a block-sparse attention mechanism that uses coarse-to-fine pyramid Top-K selection with LogSumExp scoring and hardware-aware Triton kernels to reduce block-selection complexity from O(N²/C) to O(N log N); across 418M, 1.47B, and 2.67B scales it matches BSA, NSA, and HiLS on language modeling and commonsense reasoning, achieves the highest average accuracy among sparse methods on six containment tasks, and speeds up block selection over BSA by 2.86x, 5.31x, and 9.95x at 64K, 128K, and 256K.

Researchers from Shanghai Jiao Tong University and ByteDance Seed propose PISA, a block-sparse attention mechanism that uses coarse-to-fine pyramid Top-K selection with LogSumExp scoring and hardware-aware Triton kernels to reduce block-selection complexity from O(N²/C) to O(N log N); across 418M, 1.47B, and 2.67B scales it matches BSA, NSA, and HiLS on language modeling and commonsense reasoning, achieves the highest average accuracy among sparse methods on six containment tasks, and speeds up block selection over BSA by 2.86x, 5.31x, and 9.95x at 64K, 128K, and 256K.