DETER moves suppression into pretraining: a 1B model needs over 20 times more PubMed fine-tuning tokens to match an unfiltered model
Synopsis
Once an open-weight model is released, an attacker can fine-tune it on target-domain data and quickly undo post-training unlearning. DETER interleaves suppression updates with ordinary pretraining: a document classifier first removes medical documents from FineWeb-Edu, then gradient ascent on a subset of the removed documents actively lowers the model's probability for that text. With medicine as a proxy target domain across four models from 113M to 1030M parameters, reaching the unfiltered model's PubMed loss takes more than twice the fine-tuning tokens of filtering alone; at 1030M, DETER needs 2.09M PubMed tokens versus 0.10M for filtering alone.
Interpretation
Suppression is moved into pretraining and interleaved with ordinary pretraining, so a released model is harder to retrain on the suppressed medical knowledge through adversarial fine-tuning. Earlier approaches either unlearned after pretraining or only filtered target-domain documents out of the corpus; the former is reversed by little fine-tuning, and the latter gives no explicit signal about how the model should behave on the target domain. Across the 113M, 224M, 521M and 1030M models, reaching the unfiltered model's PubMed loss takes more than twice the fine-tuning tokens of filtering alone and also exceeds the post-hoc unlearning method RMU; RMU's medical loss falls steeply early in fine-tuning and approaches the filtering curve.
The method pretrains normally on the filtered corpus and periodically inserts a few suppression steps whose objective is the negative next-token loss on the removed documents. Negative next-token loss is gradient ascent, which is often reported to degrade general capabilities in post-hoc unlearning; interleaved with pretraining, this simple objective causes limited degradation of retained capabilities. Pretraining and suppression share the same model parameters but keep separate optimizer states; at 113M, sharing optimizer statistics makes suppression look stronger at the end of pretraining, but nearly all the extra medical loss disappears within the first few adversarial fine-tuning steps and spillover into biology grows.
Perspective
The conclusions rest on models of up to 1030M parameters with medicine as a proxy target domain, with recovery difficulty measured by held-out PubMed loss and an attack budget at the largest scale of 0.010% of pretraining tokens. The intended readers are researchers and engineers concerned with capability control in open-weight models and with pretraining design.
Every method reaches the unfiltered baseline's PubMed loss within the evaluated attack budget, so the findings establish only that more target-domain fine-tuning is required under these attacks and model sizes. Classifier quality and suppression-set construction are not systematically varied, leaving the effects of false positives, missed subdomains and the source of suppression data unresolved; the evaluation splits are also defined by the same classifier and may share blind spots with training-data selection.