# Continual Pre-training

3 related articles

Subscribe to updates

arXiv Nov 6, 2026
SLA moves spatial alignment into continual pre-training: 83.8% average success across eight real deployment conditions, 24.0 points over the baseline
SLA extends spatial alignment for vision-language-action models from task post-training into continual pre-training, and keeps it during downstream post-training. Initialized from a pretrained policy, it jointly optimizes action prediction and a cosine alignment between intermediate visual embeddings and spatial representations from a frozen 3D perception model, using feature-dependent gates to aggregate multiple VLA layers and probabilistic state dropout to reduce reliance on demonstration trajectories. It reaches 72.3% on the RoboTwin 2.0 Clean2Random setting and 83.8% average success across eight real deployment conditions, 24.0 percentage points above the baseline.
SLA extends spatial alignment for vision-language-action models from task post-training into continual pre-training, and keeps it during downstream post-training. Initialized from a pretrained policy, it jointly optimizes action prediction and a cosine alignment between intermediate visual embeddings and spatial representations from a frozen 3D perception model, using feature-dependent gates to aggregate multiple VLA layers and probabilistic state dropout to reduce reliance on demonstration trajectories. It reaches 72.3% on the RoboTwin 2.0 Clean2Random setting and 83.8% average success across eight real deployment conditions, 24.0 percentage points above the baseline.
Beijing Academy of Artificial Intelligence Tsinghua University The Hong Kong University of Science and Technology (Guangzhou)
Fuhao Li（First author）
SLA Continual Pre-training Vision-Language-Action Models Implicit Spatial Alignment
arXiv Oct 8, 2026
Raising Adam's second moment only for the output projection removes 39.4%–67.9% of catastrophic forgetting across eight settings from 160M to 12B
In data-free continual pre-training and fine-tuning, the authors use training-time parameter freezing to localize catastrophic forgetting to the output embedding rows of tokens rare in the new corpus, show that these rows receive persistent one-sided softmax gradients that Adam's second-moment normalization amplifies into full-sized updates, and raise Adam's second moment only for the output projection, removing 39.4%–67.9% of forgetting across seven stable settings from 160M to 12B and four model families without degrading target learning.
In data-free continual pre-training and fine-tuning, the authors use training-time parameter freezing to localize catastrophic forgetting to the output embedding rows of tokens rare in the new corpus, show that these rows receive persistent one-sided softmax gradients that Adam's second-moment normalization amplifies into full-sized updates, and raise Adam's second moment only for the output projection, removing 39.4%–67.9% of forgetting across seven stable settings from 160M to 12B and four model families without degrading target learning.
Seoul National University of Science and Technology
Jonghyun Han（First author）
Catastrophic Forgetting Output-scoped epsilon Continual Pre-training Large language models
arXiv Oct 5, 2026
Fisher-guided submodular data selection lets 1B selected tokens beat 10B replay tokens on both adaptation and forgetting in medical continual pre-training
The work shows that loss-based data selection drives the post-continual-pre-training Fisher diagonal downward on exactly the high-Fisher coordinates the pretrained model had committed to while leaving low-Fisher coordinates largely untouched, and proposes a Fisher-aware selector that decomposes each candidate's gradient into an anchor component and a frontier component and aggregates them with a log-determinant submodular objective optimized in a single streaming pass; on TinyLlama-1.1B and Llama-3.1-8B continual pre-training over medical data it improves target-domain quality while bounding forgetting on held-out pretraining benchmarks, and 1B selected tokens outperform a replay strategy trained with 10B tokens on both adaptation and forgetting, a roughly 10x token-efficiency advantage.
The work shows that loss-based data selection drives the post-continual-pre-training Fisher diagonal downward on exactly the high-Fisher coordinates the pretrained model had committed to while leaving low-Fisher coordinates largely untouched, and proposes a Fisher-aware selector that decomposes each candidate's gradient into an anchor component and a frontier component and aggregates them with a log-determinant submodular objective optimized in a single streaming pass; on TinyLlama-1.1B and Llama-3.1-8B continual pre-training over medical data it improves target-domain quality while bounding forgetting on held-out pretraining benchmarks, and 1B selected tokens outperform a replay strategy trained with 10B tokens on both adaptation and forgetting, a roughly 10x token-efficiency advantage.
University of Illinois Chicago
Zhenghao Zhao（First author）
Data Selection Continual Pre-training Catastrophic Forgetting Large language models

You’re all caught up

SLA moves spatial alignment into continual pre-training: 83.8% average success across eight real deployment conditions, 24.0 points over the baseline

Raising Adam's second moment only for the output projection removes 39.4%–67.9% of catastrophic forgetting across eight settings from 160M to 12B

Fisher-guided submodular data selection lets 1B selected tokens beat 10B replay tokens on both adaptation and forgetting in medical continual pre-training