LoRA stops giving every token a fixed rank: noise-floor squelch activates 8.6 of 32 components

Low-rank adaptation applies every rank-one component of its update to every token, and mixture-of-experts variants replace this with a learned router that activates a fixed number of components; SQUELCH instead lets the active rank vary per token. Earlier practice either applied all rank-one components to every token or had a router activate a fixed count. On commonsense reasoning, knowledge, math, code and GLUE with LLaMA-3.1-8B, Qwen3 and DeBERTaV3, SQUELCH improves over LoRA by 2.2 to 3.2 points and over the strongest mixture baseline by 0.8 to 1.2 points, activating 8.6 of 32 components per token on average.

It reads a token's responses to the rank-one components as a few large values on a Gaussian noise floor whose width matches an untrained adapter, with the floor carrying cross-talk rather than adaptation. SQUELCH adds no trainable parameters to LoRA and requires no rank budget to be specified. The threshold multiple is chosen per module by Stein's unbiased risk estimate, after which one explaining-away step on the value Gram matrix suppresses redundant co-active components and recruits complementary ones.

This report covers the summary of the work, spanning evaluations on commonsense reasoning, knowledge, math, code and GLUE with LLaMA-3.1-8B, Qwen3 and DeBERTaV3. It is aimed at researchers and engineers working on parameter-efficient fine-tuning and mixture-of-experts routing.

The material is a summary and does not give per-task breakdowns, the specific threshold values, or the cost of the explaining-away step; the average of 8.6 active components is a reported mean, and the per-token distribution is not described.