EWC stops only guarding old weights: FS-EWC shapes the solution first, then protects it, beating EWC across LoRA benchmarks
Synopsis
In replay-free continual learning, EWC penalizes changes to parameters deemed important using Fisher information estimated at the retained solution, but that local estimate does not capture sensitivity after the parameters move. FS-EWC uses Fisher information earlier, shaping the current task's solution by minimizing the worst-case Fisher trace over a neighborhood, and builds the retained anchor by averaging checkpoints along the optimization trajectory. Across vision and language benchmarks, FS-EWC generally improves on the corresponding EWC baseline under both full-parameter fine-tuning and LoRA.
Interpretation
Forgetting in continual learning depends not only on how a solution is protected afterward but on whether the protected solution itself tolerates later parameter drift; FS-EWC extends Fisher information from parameter protection to solution formation. EWC and its online, trajectory-aware and structured-approximation extensions mainly refine how historical importance is constructed and used, while still treating the solution to be protected as given. On the four vision benchmarks CIFAR-100, ImageNet-R, ImageNet-A and DomainNet, FS-EWC improves final average accuracy over the corresponding EWC baseline under both full-parameter fine-tuning and LoRA.
During training, FS-EWC adds a neighborhood Fisher-trace penalty to the EWC objective so the current solution is less sensitive to bounded future displacement within a radius; at task completion the retained anchor is the average of selected checkpoints along the optimization trajectory, and Fisher is then estimated at that anchor for later consolidation. The gradient of the worst-case neighborhood Fisher trace carries a Hessian-vector product correction, and the practical solver approximates the inner maximization with a finite-difference probe plus a single normalized ascent step. On the five fine-grained and heterogeneous vision benchmarks Aircraft, Cars, CUB-200, Flowers and 5Datasets, FS-EWCLoRA improves final average accuracy over the matched EWCLoRA baseline in every case.
Perspective
Experiments cover CIFAR-100, ImageNet-R, ImageNet-A, DomainNet and five fine-grained vision benchmarks on ViT-B/16, plus continual text classification on Llama-3.2-1B and Llama-3.2-3B; all are replay-free class-incremental settings where historical data are neither retained nor replayed.
The tolerance radius in the controlled displacement diagnostic is evaluated only on a measured grid, making it an empirical summary rather than a guarantee over the continuous neighborhood; Llama-3.2-3B under Order 3 ends slightly below EWCLoRA in final average accuracy; Kronecker-factored EWC hit numerical instability and produced NaN at one historical-regularization coefficient.