Language models pick their own replay targets: merging two task banks lifts held-out retention by up to 4.42 points
Synopsis
Fine-tuning a language model on a new task can erode old abilities, and the prior data kept for protection covers only part of what the model knows. Researchers organize task descriptions and examples into Task Banks, let a frozen pre-update model generate its own replay, and test protection on four held-out tasks excluded from bank construction. Merging two partial banks raises mean held-out retention above either bank alone at all three tested sizes, by up to 4.42 percentage points; at 4B, selection informed by the incoming task beats random selection by 2.75 points, an advantage absent at smaller sizes.
Interpretation
Fine-tuning on a new task shifts performance on prior tasks, and average accuracy can hide that shift. The data previously used to protect old abilities covers only part of what the model knows, while forgetting can reach beyond it. Averaged over four held-out tasks and two seeds, 3.56% of examples flip from correct to incorrect and 3.61% from incorrect to correct, leaving about a one-point net change, so paired retention is reported alongside task accuracy.
Replay generated by the frozen pre-update model protects behavior outside the supplied targets. Rehearsing a single supplied task limits protection to that task. In the C-STANCE comparison, broad replay gives better mean held-out accuracy change than single-task replay while incoming-task gains stay close.
Combining two partial Task Banks broadens the replay inventory and raises mean held-out retention. A single partial bank covers only part of prior behavior, and its protection varies by held-out task. Across the three tested sizes, the combined bank exceeds the better partial bank by 4.42, 3.32, and 2.99 percentage points, with incoming gains 0.45, 0.60, and 0.28 points below SFT.
The model can also help choose which targets to replay, not only generate replay data. Random selection uses no information about the incoming task. At 4B with only two bank entries, incoming-aware selection improves mean held-out retention by 2.75 points over random selection with a 0.04-point higher incoming gain, staying within 0.32 points of the full four-entry bank; no such advantage appears at 0.6B or 1.7B.
Perspective
The report covers Qwen3-0.6B, 1.7B, and 4B plus Qwen3.5-4B for forgetting diagnostics, with MedQA and C-STANCE as incoming tasks, MMLU, HellaSwag, GSM8K, and BoolQ in the Task Bank, and ARC-C, WinoGrande, PIQA, and SVAMP held out from replay construction and target selection. It suits researchers and engineers working on continual learning and fine-tuning forgetting. Most studies perform one update, and the two-stage, one-seed pilot has mixed outcomes, so long-run accumulation is not established.
The selection advantage appears only in the tested 4B configuration, with no corresponding advantage at smaller sizes, and learner, selector, and replay-generator strength are not varied independently. Under a fixed example allocation, replacing incoming examples with replay improves preservation at the cost of smaller new-task gains, and that comparison matches nominal rows and optimizer steps while token exposure and generation cost differ. Task-Bank Self-Replay still depends on supplied preservation targets, replay can reproduce model errors, and the bank-aware expansion pilot is a single-arm, single-operating-point test that does not estimate the incremental preservation value of added targets over a fixed bank.