LSL gates weight updates to the current data's support with a GMM, letting a 7B-parameter LLM keep 96.6% of pretrained capability after multi-phase finetuning

Starting from the input-output mapping of a single weight matrix, the paper shows that updates produced by gradient-based optimizers act on all inputs: any prior input not orthogonal to the update has its logits altered, making such updates suboptimal under a natural retention objective in which updates should affect only the support of the current phase's distribution. Prior work either relies on buffers of previous data, data-dependent regularization, or orthogonality constraints, or protects only previous finetuning phases, or approximates the orthogonal complement of pretraining data from the weights; this work recasts forgetting as a per-matrix input-space support problem and reports that the orthogonal-complement approximation is insufficient. A two-dimensional six-class toy classification problem reproduces catastrophic forgetting, with a derivation showing the logit change depends on a dot product, and a note that Adam and Muon still produce matrix updates acting on all inputs; the insufficiency of the orthogonal-complement approximation is analyzed in Appendix C.

During a new learning phase, LSL pairs two components: a weight adapter trained as usual to minimize the loss, and a gating function that estimates the support of the current data in the adapter's input space; at inference the gate applies the adapter only to tokens whose activations fall within that support. Unlike standard finetuning with an always-open gate, or methods that restrict updates to low-rank or orthogonal subspaces, LSL delegates locality to the gate so that the update takes effect only near its training distribution. Training and inference algorithms are provided (Alg. 1, Alg. 2), and inference is described as fully parallelizable during prefill just like standard MLPs.

The gate is implemented with Gaussian Mixture Models: one GMM is fit to the current phase's distribution and a second to a small sample from a generic pretraining dataset, and each input is assigned by larger likelihood, avoiding manual threshold tuning; Gaussian density decays exponentially with distance, giving the gate an inductive bias to stay closed on unseen inputs. The paper provides a theoretical connection: against the worst-case test distribution a gate's error decomposes into deficit and excess, the training objective optimizes the deficit directly but is blind to the excess, and density estimators can recover optimal gates under explicit conditions; the input-dependent threshold form satisfies the corresponding bound. Appendix E gives the full derivation, including comparisons of local estimators, density estimation, and classifiers plus three one-dimensional counterexamples; empirically the gate opens on about 16% of tokens from out-of-distribution pretraining tasks (5% with temporal smoothing) while retaining 96.6% of pretrained performance (98.8% with smoothing).

Evaluated on Qwen2.5-7B-Instruct for cybersecurity instruction tuning, low-resource English-to-Igbo translation, and chemistry instruction tuning, LSL achieves strong new-task performance while maintaining close-to-optimal retention of pretrained capabilities on GSM8K, HumanEval, and IFEval; in sequential multi-phase finetuning, LSL retains pretrained capabilities and each finetuned task is preserved after its phase ends. LoRA forgets in all settings, OP-LoRA performs very similarly to LoRA (Appendix C shows most of the update's energy already lies in the orthogonal subspace), and LwF beats the weight-based methods but still trails LSL; the LoRA baseline drops zero-shot performance on task 3 by 11% after task 1, task 1 by 8% after two phases, and task 2 by 5% after one phase. Each experiment averages over three seeds, hyperparameters are selected on new-task validation performance with retention benchmarks never used for selection, and the multi-phase experiment caps evaluation samples per benchmark at 300 due to long runtimes.

The result targets streaming multi-phase finetuning where prior data is unavailable and per-phase memory is bounded, and applies to pretrained LLMs with good representations; the gate assumes each phase's activations can be captured by a GMM, so it suits base models with high-quality representations, and the scaling experiment shows larger models retain better. The method integrates with standard adapter finetuning, gates are fit independently per module for easy parallelization, and the reference dataset need not relate to the original pretraining data, requiring no more than 1M tokens empirically. The authors note future work on larger models, other modalities, reinforcement learning objectives, and hundreds of phases, the latter potentially enabling test-time training that stores new knowledge in weights rather than context.

The theoretical motivation concerns a single weight matrix, and its extension to the full multi-layer network is supported empirically but not formally. The gate relies on the quality of the base model's representations, so retention may be limited if a phase's activations are not well captured by a GMM. Adapters cannot be merged into the base weights, so inference cost grows linearly with the number of phases, though it is highly parallelizable to trade compute for inference time. The gate still opens on about 16% of tokens from out-of-distribution pretraining tasks (5% with smoothing), indicating support estimation is a practical approximation rather than an exact threshold. The multi-phase experiment caps evaluation samples per benchmark at 300, and the GMM fitting step may need further acceleration when scaling to large finetuning datasets. This is a full-text read, but figures and tables are presented as textual descriptions, so exact curve values require consulting the original figures.