GRPO safety alignment loses its signal on hard prompts: Beacon restores a safe direction and cuts average safety error by up to 32.83 points across four models
Synopsis
Safety alignment of large reasoning models commonly relies on GRPO, but responses sampled for one prompt often share the same safety outcome, so the within-group relative advantage vanishes and the model gets no direction toward safety on persistently unsafe prompts. Beacon pairs safety signal recovery, which filters high-contrast trajectories and directly supervises a safe reference response when the signal collapses, with safety-utility decoupled alignment, which assigns category-aware rewards and applies KL anchoring only to benign prompts. Across four Qwen3 and DeepSeek-R1-Distill variants and eight benchmarks, average safety error falls by 17.69 to 32.83 percentage points relative to the initial policies, with XSTest over-refusal between 0.40% and 1.20%.
Interpretation
In safety alignment training, GRPO's within-group relative advantage fails in two ways: when all sampled responses to a prompt are safe or all are unsafe, the group rewards are identical, the relative advantage is zero, and the policy gradient carries no safety direction; on persistently unsafe prompts, entropy regularization then spreads probability mass over more unsafe responses, producing diversification of unsafe behavior. Earlier work applying GRPO to reasoning-model safety alignment focused mainly on data construction and reward design, and left the group-relative optimization itself less examined. Running vanilla GRPO on Qwen3 and DeepSeek-R1 models with harmful prompts and binary safety rewards showed many prompts forming all-unsafe groups with zero relative advantage, while the unsafe response rate stayed high and semantic diversity increased.
Beacon consists of safety signal recovery, which keeps a safety-directed learning signal on harmful prompts, and safety-utility decoupled alignment, which keeps safety optimization from spilling onto benign requests as over-refusal. Safety signal recovery combines advantage-guided trajectory filtering, which ranks rollouts by within-group advantage gap and keeps only the highest-contrast trajectory pairs for the policy update, with safe reference supervision, which directly maximizes the autoregressive likelihood of a given safe reference response, including its reasoning trace, final answer and end-of-sequence token, when the group signal collapses. Adding components progressively on Qwen3-8B, safe reference supervision raised the safe-response share on harmful prompts from 24.34% to 49.74% and cut average safety error from 5.16% to 2.36%; adding category-aware rewards and benign-only KL anchoring then lowered XSTest over-refusal from 14.40% to 1.20% and raised reasoning accuracy from 78.89% to 80.64%.
Across the four model variants, Beacon reaches the lowest average safety error and the highest average reasoning accuracy while holding over-refusal at a low level. Relative to the initial policies, average safety error drops by 17.69 to 32.83 percentage points and reasoning accuracy rises by 0.26 to 4.56 percentage points. On DeepSeek-R1-Distill-8B, GRPO shows 16.12% safety error, 63.33% reasoning accuracy and 25.60% over-refusal, while Beacon shows 6.27%, 67.94% and 0.40%; on Qwen3-8B against DPO, GRPO, SFT+GRPO, GDPO, NGRPO and MDP-GRPO, Beacon reaches 1.88% average safety error and 80.64% average reasoning accuracy.
Perspective
The demonstrated scope is safety and reasoning evaluation on four Qwen3 and DeepSeek-R1-Distill variants across eight benchmarks, with safety error judged by Llama-Guard-3 on single generations and over-refusal measured on the XSTest safe subset with WildGuard; training data comes from ThinkSafe, whose safe reference responses also serve as the targets for safe reference supervision.
The method relies on existing reward models and safety evaluators to supply safety, utility and honesty signals, and its effectiveness may depend on the quality and reliability of those signals; the work focuses on safety alignment under harmful and benign prompt settings, leaving broader alignment objectives such as factuality, uncertainty calibration and long-horizon interactions uncovered.