Haifang Jian

Query-conditioned sparse KV selection must wait for the target-layer query before scoring candidates and generating indices, so routing sits on the immediate pre-attention path. EarlySelectKV replaces the routing query with a proxy built from an earlier hidden state, precomputing the KV indices the next layer will access while final attention still uses the real query. On RocketKV it differs by +0.018 points on LongBench, about −0.01 points on RULER and +0.25 percentage points on representative NIAH/Passkey settings; in a TensorRT-LLM deployment with CPU-offloaded routing, end-to-end output throughput rises by up to 12.4% and GPU-resident memory falls by 11.1% of the combined KV/K-summary pool.

Query-conditioned sparse KV selection must wait for the target-layer query before scoring candidates and generating indices, so routing sits on the immediate pre-attention path. EarlySelectKV replaces the routing query with a proxy built from an earlier hidden state, precomputing the KV indices the next layer will access while final attention still uses the real query. On RocketKV it differs by +0.018 points on LongBench, about −0.01 points on RULER and +0.25 percentage points on representative NIAH/Passkey settings; in a TensorRT-LLM deployment with CPU-offloaded routing, end-to-end output throughput rises by up to 12.4% and GPU-resident memory falls by 11.1% of the combined KV/K-summary pool.