Swap Adam for STABLE and old tasks stop collapsing: the same LoRA goes from 0.66% to 73.12% average accuracy on a 100-task stream
Synopsis
Forgetting in continual learning is usually fought with replay, regularization or architectural changes; STABLE instead works inside the optimizer, treating the distance from the previous task's solution as a budget that tightens the pullback when exceeded and relaxes it once drift returns inside. On the 100-task ImageNet-R stream the same LoRA goes from 0.66% average accuracy with Adam to 73.12%, with average forgetting falling from 60.22% to 10.17%, and CLIP zero-shot transfer stays near its pre-trained level.
Interpretation
Forgetting in continual learning can be handled at the optimizer level: STABLE constrains how far parameters move from the solution reached at the end of the previous task, without changing the data stream, the loss or the parameter-efficient tuning architecture. Earlier continual-learning methods mostly rely on replaying old samples, regularizing important parameters, or isolating capacity per task with prompts, routing or adapters, which adds training complexity and sometimes inference cost. On the 100-task ImageNet-R stream, the same LoRA architecture reaches 0.66% final average accuracy and 60.22% average forgetting with Adam, versus 73.12% and 10.17% with STABLE; on the 50-task CIFAR-100 stream accuracy rises from 1.30% to 88.77%.
The mechanism stores the previous task's parameters as an anchor at each task boundary and uses a primal-dual update to tune a pullback term that keeps training from straying too far from that anchor. The pullback strength is set online by a dual variable rather than by a coefficient fixed before training as in EWC, SI, MAS or L2-SP; the dual variable rises when drift exceeds the budget and decays multiplicatively once drift returns inside it. In the 100-task ImageNet-R LoRA run, distance from the anchor rises sharply in the early epochs after the switches at tasks 10, 50 and 100, the dual variable rises once the budget is exceeded and draws the trajectory back, then decays; Adam's drift grows without constraint.
The drift budget is calibrated once on the first task and then reused for the whole stream. The budget is set from the 75th percentile of update-magnitude statistics recorded during task 1, so later tasks need no per-dataset tuning. ImageNet-A-100 classes range from 26 images to several times that, so its tasks differ in difficulty and training-set size; with the budget calibrated from its first task, LoRA with STABLE still reaches 62.34% average accuracy and 12.11% average forgetting on that stream.
Perspective
Experiments cover frozen ImageNet-21K pre-trained ViT-B/16 and CLIP ViT-B/16 backbones, training only parameter-efficient modules such as LoRA or prefix prompts, with 100-, 50- and 50-task unimodal streams and 10-task multimodal streams, and inference through a covariance-whitened nearest-class-mean head or the CLIP similarity head. Full-model finetuning, training from scratch and other modalities are outside the scope.
The prediction-preservation guarantee and margin-erosion analysis apply only to distance-based heads induced directly by the representation; other heads and generative objectives are outside the current theory and experiments. The budget is calibrated from task 1, and a deliberately atypical first task has not been tested; combining STABLE with specialized methods such as InfLoRA, HiDeLoRA or CODA-Prompt is left for future work.