Low-rank compression stops spending the same compute per token: training only routers lifts Llama-2-7B downstream accuracy by 7.6 points
Low-rank compression previously assigned a fixed rank during inference, so every token cost the same compute. LRCC adds one lightweight router per Transformer block that selects among a few nested low-rank paths for each token, training only the routers while the low-rank factors stay frozen. Within the same average active-parameter budget, Llama-2-7B gains 7.6 percentage points in average downstream accuracy over static low-rank compression; at matched batch-size-1 decoding latency, Llama-3.2-1B improves both perplexity and downstream accuracy and Llama-2-7B stays competitive, without specialized CUDA kernels.