Luxi He

Undesired behaviors in language models can re-emerge during later fine-tuning, and standard post-hoc suppression had not been compared on immediacy, durability and capability together. PEARL adds adversarial prefixes, continuations from intermediate states of cached misaligned trajectories, to GRPO's on-policy rollouts so the policy practices recovering from partly wrong states. Across Qwen3-4B and GPT-OSS-20B and two organism types, PEARL suppresses strongly while preserving task accuracy, with the clearest gains on the broad-trigger organism and GPT-OSS-20B; on Qwen3-4B with the narrow-trigger organism, SFT is more durable under heavy targeted retraining.

Undesired behaviors in language models can re-emerge during later fine-tuning, and standard post-hoc suppression had not been compared on immediacy, durability and capability together. PEARL adds adversarial prefixes, continuations from intermediate states of cached misaligned trajectories, to GRPO's on-policy rollouts so the policy practices recovering from partly wrong states. Across Qwen3-4B and GPT-OSS-20B and two organism types, PEARL suppresses strongly while preserving task accuracy, with the clearest gains on the broad-trigger organism and GPT-OSS-20B; on Qwen3-4B with the narrow-trigger organism, SFT is more durable under heavy targeted retraining.