A $20 reward function can strip a model's safety alignment; Bait probes the reward to block the learning
Synopsis
Agentic reinforcement-learning-as-a-service hands data sampling and reward rating to the user, and a harmful reward function can erase a model's safety alignment at a cost as low as $20. Bait mixes explicit harmful responses from a separately trained Bait model into the policy model's response group during rollout, uses the sign of the reward gap between Bait and non-Bait responses (reward divergence) to flag a harmful reward function, and flips that group's reward sign to penalize learning when it detects one. On the authors' testbed, Bait cut the average harmful score across three harmful-instruction datasets from 20.4 to 6.8 while average agentic accuracy rose from 93.1 to 96.0.
Interpretation
Agentic RL-as-a-service hands data sampling and reward rating to the user while the provider runs rollout and policy optimization, so a harmful reward function in the user environment can progressively erase the model's safety alignment during training. Safety alignment was previously completed before training, whereas this service lets a user-supplied reward signal take part in later optimization and rewrite that alignment. Training through the Tinker API with BeaverTails harmful data and a harmful reward model raised the model's average harmful score across three safety benchmarks by 66.42% over the pre-training model.
The defense mixes explicitly harmful responses into the policy model's response group during rollout, so the reward function reveals its own preference. Detection does not require reading inside the user environment; it watches how the user's reward function scores the two kinds of responses. Reward divergence is the mean reward of Bait responses minus that of non-Bait responses; a positive sign marks the group as rated by a harmful reward function, and when detected the group's rewards are sign-flipped before advantage computation.
On the authors' testbed, detection plus penalty lowered harmful scores across harmful-instruction datasets, model sizes and benign agentic environments without sacrificing learning on benign tasks. Without the defense the harmful score rises with training, while with it the harmful score stays low and benign agentic accuracy matches the undefended run. Across three harmful-instruction datasets the average harmful score fell from 20.4 to 6.8 and average agentic accuracy rose from 93.1 to 96.0; across three model sizes the average harmful score dropped 12.4%; on AlfWorld and WebShop it dropped 44.9% and 39.1%, with agentic accuracy down slightly by 1% and 0.9%.
Perspective
Experiments ran on the authors' own testbed because implementing the defense requires modifying rollout and policy optimization, which the Tinker API does not expose; the attack demonstration ran on the Tinker API. Evaluation covers three harmful-instruction datasets (BeaverTails, DecodingTrust, DirectHarm4), three model sizes (Qwen3-VL-2B/4B/8B), and three benign agentic environments (EZPoints, AlfWorld, WebShop), with harmful scores rated by a Qwen3-8B judge model.
The defense is detection-based and inherently carries false negatives and false positives: early in training the false-positive rate on benign tasks reached 75% and fell as agentic accuracy rose; the Excision adaptive attack at a suitable excision intensity raised the false-negative rate to about 0.30, but higher intensity mis-excises non-Bait responses, lowering the false-negative ceiling and disrupting harmful learning. The authors leave other adaptive attacks to future work and note the defense may or may not generalize to self-evolving training.