FOM-UL unlearns by editing only a few layers: under 4-bit quantization its residual memorization stays below whole-model baselines with higher retain utility
Synopsis
Unlearning a large language model no longer requires diffuse updates across the whole model. FOM-UL scores each transformer layer by the ratio of its forget-set gradient norm to its retain-set gradient norm, updates only the few high-scoring layers, and adds layers one at a time when forgetting falls short. Across TOFU, KnowUnDo and MUSE-style evaluations it leaves less residual memorization than GA, NPO and KLD baselines while keeping retain-set utility close to the vanilla model, and it holds a favorable forgetting-utility trade-off under 8-bit and 4-bit post-training quantization, though residual memorization does rise under 4-bit relative to full precision.
Interpretation
Unlearning a large language model can now be confined to a few layers instead of a diffuse update across the whole model. Earlier gradient-ascent, negative-preference-optimization and KL-constrained methods typically apply updates model-wide, spreading the forgetting signal, and those updates can be erased by low-bit rounding under post-training quantization. On TOFU-World Facts with Llama-3.2-1B, FOM-UL-Full lowers residual memorization M1 from 5.80±0.07 for the vanilla model to 1.24±0.03, while retain-set utility M4 is 2.90±0.03, level with the vanilla model's 2.90±0.03.
Layer choice comes from a forget-to-retain significance score, and updates land only on the high-scoring layers. The score is the ratio of a layer's forget-set gradient norm to its retain-set gradient norm, so a high ratio marks a layer that responds strongly to the forgetting objective with little interference on the retain objective. On TOFU-World Facts with Llama-3.2-1B, selecting a single high-scoring late layer gives M1 of 2.35 and M4 of 2.85, whereas selecting only the first four layers gives M1 of 4.80 and M4 of 2.05.
Concentrating updates in a few layers makes forgetting harder to round away under low-bit quantization. Diffuse model-wide updates are small in magnitude and can fall back into the same quantization bin, letting suppressed knowledge reappear. Under 4-bit quantization on Llama-3.2-1B, FOM-UL reports M1 of 2.44 and M4 of 2.04, while SURE+NPO in the same setting reports M1 of 2.42 and M4 of 1.80.
Perspective
Evaluations cover TOFU-World Facts, KnowUnDo and the BOOKS and NEWS subsets of MUSE, on Llama-2 7B, GPT-2, Llama-3.2-1B, Gemma-3-1B and Qwen3.5-4B, and are checked under FP32, 8-bit and 4-bit quantization and jailbreak prompt wrappers. The authors leave formal guarantees of complete erasure to future work.
The authors note that effectiveness depends on the reliability of layer-attribution signals, which can vary across prompts, domains and evaluation setups; highly entangled or redundantly encoded knowledge may require expanding the updated layer set or more iterations, raising compute and introducing utility trade-offs. They also state that FOM-UL's negative M3 under 4-bit quantization indicates deviation from the retrained privacy baseline rather than improved privacy, and that 4-bit quantization generally raises residual memorization relative to 8-bit.