# Catastrophic Forgetting

Forgetting in continual learning is usually fought with replay, regularization or architectural changes; STABLE instead works inside the optimizer, treating the distance from the previous task's solution as a budget that tightens the pullback when exceeded and relaxes it once drift returns inside. On the 100-task ImageNet-R stream the same LoRA goes from 0.66% average accuracy with Adam to 73.12%, with average forgetting falling from 60.22% to 10.17%, and CLIP zero-shot transfer stays near its pre-trained level.

Forgetting in continual learning is usually fought with replay, regularization or architectural changes; STABLE instead works inside the optimizer, treating the distance from the previous task's solution as a budget that tightens the pullback when exceeded and relaxes it once drift returns inside. On the 100-task ImageNet-R stream the same LoRA goes from 0.66% average accuracy with Adam to 73.12%, with average forgetting falling from 60.22% to 10.17%, and CLIP zero-shot transfer stays near its pre-trained level.

Dense Transformer layers are expanded into sparse mixture-of-experts layers, giving law and biomedicine their own full MLP experts while a router is supervised with a small number of examples to pick the right expert per domain. On Llama-3.1-8B and Qwen-3-8B the method posts the highest biomedical and legal averages and the highest general-domain average; with only 256 examples per domain for router supervision, routing accuracy on specialized domains reaches 90–98% after adaptation, against 43–70% for routers trained without it.

Dense Transformer layers are expanded into sparse mixture-of-experts layers, giving law and biomedicine their own full MLP experts while a router is supervised with a small number of examples to pick the right expert per domain. On Llama-3.1-8B and Qwen-3-8B the method posts the highest biomedical and legal averages and the highest general-domain average; with only 256 examples per domain for router supervision, routing accuracy on specialized domains reaches 90–98% after adaptation, against 43–70% for routers trained without it.

Progressive neural networks transfer knowledge by keeping every frozen earlier column, so stored state and inference grow with each generation. WeanNet keeps a single frozen parent, feeds its layer-wise activations to a fresh student through a scalar gain, and decays that gain linearly to zero during training, so the parent can be removed at deployment. On a concept-drift task it found the new correct action on drifted states more often than the architecturally matched PNN-Reset control at every tested entropy setting, though it also led on never-drifted states and stayed comparable to training from scratch.

Progressive neural networks transfer knowledge by keeping every frozen earlier column, so stored state and inference grow with each generation. WeanNet keeps a single frozen parent, feeds its layer-wise activations to a fresh student through a scalar gain, and decays that gain linearly to zero during training, so the parent can be removed at deployment. On a concept-drift task it found the new correct action on drifted states more often than the architecturally matched PNN-Reset control at every tested entropy setting, though it also led on never-drifted states and stayed comparable to training from scratch.

Adult endotracheal tube AI models have been hard to apply to newborns, where annotated imaging is scarce and privacy rules block data sharing. An international retrospective study across 30 neonatal intensive care units in 20 countries and territories used continual learning: the model passes from hospital to hospital, and each site retrains it on 50 local radiographs inside its own environment without sharing patient data. The final model averaged 0.67 cm combined endotracheal tube tip and carina localization error, versus 1.29 cm for hospital-specific fine-tuning and 4.30 cm for the adult model used directly, improving localization at 29 of 30 sites and deep-intubation detection at 27 of 30.

Adult endotracheal tube AI models have been hard to apply to newborns, where annotated imaging is scarce and privacy rules block data sharing. An international retrospective study across 30 neonatal intensive care units in 20 countries and territories used continual learning: the model passes from hospital to hospital, and each site retrains it on 50 local radiographs inside its own environment without sharing patient data. The final model averaged 0.67 cm combined endotracheal tube tip and carina localization error, versus 1.29 cm for hospital-specific fine-tuning and 4.30 cm for the adult model used directly, improving localization at 29 of 30 sites and deep-intubation detection at 27 of 30.

Fine-tuning a language model on a new task can erode old abilities, and the prior data kept for protection covers only part of what the model knows. Researchers organize task descriptions and examples into Task Banks, let a frozen pre-update model generate its own replay, and test protection on four held-out tasks excluded from bank construction. Merging two partial banks raises mean held-out retention above either bank alone at all three tested sizes, by up to 4.42 percentage points; at 4B, selection informed by the incoming task beats random selection by 2.75 points, an advantage absent at smaller sizes.

Fine-tuning a language model on a new task can erode old abilities, and the prior data kept for protection covers only part of what the model knows. Researchers organize task descriptions and examples into Task Banks, let a frozen pre-update model generate its own replay, and test protection on four held-out tasks excluded from bank construction. Merging two partial banks raises mean held-out retention above either bank alone at all three tested sizes, by up to 4.42 percentage points; at 4B, selection informed by the incoming task beats random selection by 2.75 points, an advantage absent at smaller sizes.

The work introduces SFT-as-context, a training-free inference scheme in which the SFT model first answers the query and the parent model then generates the final answer using that response as context, acquiring fine-tuned capabilities through in-context learning while keeping its own general capabilities; across 19 parent-SFT model pairs and 11 benchmarks it stays close to the SFT models on fine-tuned capabilities (gaps of only 2.2 and 2.1 percentage points on AIME 2024 and LiveCodeBench, and 2.0 macro MAE on NutriBench-English) and within 2.2 percentage points of the parent models on general capabilities on average, and it can answer queries that need both capability types even when neither model succeeds alone.

The work introduces SFT-as-context, a training-free inference scheme in which the SFT model first answers the query and the parent model then generates the final answer using that response as context, acquiring fine-tuned capabilities through in-context learning while keeping its own general capabilities; across 19 parent-SFT model pairs and 11 benchmarks it stays close to the SFT models on fine-tuned capabilities (gaps of only 2.2 and 2.1 percentage points on AIME 2024 and LiveCodeBench, and 2.0 macro MAE on NutriBench-English) and within 2.2 percentage points of the parent models on general capabilities on average, and it can answer queries that need both capability types even when neither model succeeds alone.

The work defines each singular value's paired left and right singular vectors as a singular-vector channel, estimates each channel's adaptation benefit from current-task gradients and its forgetting cost from activation second moments on an unlabeled general-domain corpus, then selects trainable channels via knee-based cost screening, Pareto-front filtering, and Otsu thresholding; across four LLM families, eight TRACE tasks, and three task orders, SVC preserves pretrained general capabilities while achieving strong downstream continual-learning performance.

The work defines each singular value's paired left and right singular vectors as a singular-vector channel, estimates each channel's adaptation benefit from current-task gradients and its forgetting cost from activation second moments on an unlabeled general-domain corpus, then selects trainable channels via knee-based cost screening, Pareto-front filtering, and Otsu thresholding; across four LLM families, eight TRACE tasks, and three task orders, SVC preserves pretrained general capabilities while achieving strong downstream continual-learning performance.

In data-free continual pre-training and fine-tuning, the authors use training-time parameter freezing to localize catastrophic forgetting to the output embedding rows of tokens rare in the new corpus, show that these rows receive persistent one-sided softmax gradients that Adam's second-moment normalization amplifies into full-sized updates, and raise Adam's second moment only for the output projection, removing 39.4%–67.9% of forgetting across seven stable settings from 160M to 12B and four model families without degrading target learning.

In data-free continual pre-training and fine-tuning, the authors use training-time parameter freezing to localize catastrophic forgetting to the output embedding rows of tokens rare in the new corpus, show that these rows receive persistent one-sided softmax gradients that Adam's second-moment normalization amplifies into full-sized updates, and raise Adam's second moment only for the output projection, removing 39.4%–67.9% of forgetting across seven stable settings from 160M to 12B and four model families without degrading target learning.

The work proposes CPR (Critical-Point Routing), a token-level routing framework between a base model and its SFT expert derivative that trains a lightweight hierarchical router to estimate the per-token expert-call probability based on critical tokens where the base model fails but the expert succeeds, paired with an inference procedure combining momentum smoothing and threshold gating; across diverse model-domain configurations CPR achieves state-of-the-art in all settings, surpassing the SFT expert by 1.4-5.5% in domain performance while recovering its general-capability drop from 3.4-14.5% to at most 0.5%, with minimal overhead from invoking the expert on only one-third of tokens.

The work proposes CPR (Critical-Point Routing), a token-level routing framework between a base model and its SFT expert derivative that trains a lightweight hierarchical router to estimate the per-token expert-call probability based on critical tokens where the base model fails but the expert succeeds, paired with an inference procedure combining momentum smoothing and threshold gating; across diverse model-domain configurations CPR achieves state-of-the-art in all settings, surpassing the SFT expert by 1.4-5.5% in domain performance while recovering its general-capability drop from 3.4-14.5% to at most 0.5%, with minimal overhead from invoking the expert on only one-third of tokens.