GRPO safety alignment loses its signal on hard prompts: Beacon restores a safe direction and cuts average safety error by up to 32.83 points across four models
Safety alignment of large reasoning models commonly relies on GRPO, but responses sampled for one prompt often share the same safety outcome, so the within-group relative advantage vanishes and the model gets no direction toward safety on persistently unsafe prompts. Beacon pairs safety signal recovery, which filters high-contrast trajectories and directly supervises a safe reference response when the signal collapses, with safety-utility decoupled alignment, which assigns category-aware rewards and applies KL anchoring only to benign prompts. Across four Qwen3 and DeepSeek-R1-Distill variants and eight benchmarks, average safety error falls by 17.69 to 32.83 percentage points relative to the initial policies, with XSTest over-refusal between 0.40% and 1.20%.