A correct answer is not a correct derivation: GRPO gets a noise-aware group weight that keeps learning at high reward noise
Synopsis
GRPO, a common reinforcement-learning recipe for reasoning models, judges a rollout only by whether its final answer is right, so a correct answer can hide unreliable reasoning, and a single corrupted reward can strongly distort group-relative advantages when a response group is highly imbalanced. S-GRPO minimizes the expected squared error between observed and latent advantages under a reward-noise model to obtain a closed-form, noise-aware group weight. Across several mathematical reasoning benchmarks and model scales it stabilizes training and keeps making progress under controlled reward noise where standard GRPO shows little meaningful learning.
Interpretation
During reasoning training, the reward signal can now be discounted at the level of a whole response group according to how trustworthy that group looks, rather than only by whether the final answer is correct. GRPO relies on outcome rewards, so final-answer correctness is treated as a proxy for reasoning quality; when a group of responses is highly imbalanced, a single corrupted reward can strongly distort the group-relative advantages. Across multiple mathematical reasoning benchmarks and model scales, S-GRPO improves performance and stabilizes training relative to standard GRPO; under controlled reward-noise interventions its optimization dynamics are smoother, and it remains effective at high noise levels where standard GRPO shows little meaningful learning progress.
The mechanism assigns each prompt's group of responses a weight that multiplies the group's advantages. That weight comes from a reward-noise model in which observed correctness labels flip with some probability; minimizing the expected squared error between observed and latent advantages yields a closed-form weight. The implementation extends the Dr. GRPO module inside the public OAT training framework while retaining most of the pipeline, isolating the effect of the reweighting; experiments ran on eight 80 GB NVIDIA A100 GPUs with BF16 mixed precision, and a typical 500-step run for a 7B model took about 40 hours.
Perspective
The demonstrated scope is training and evaluation on mathematical reasoning benchmarks, plus supplementary experiments on the MBPP and APPS code-generation benchmarks, covering base models including Qwen2.5-Math-1.5B-Instruct, Qwen2.5-Math-7B-Base and Llama-3.2-3B-Base. The intended reader is an engineer or researcher training reasoning models who must handle noisy outcome rewards. The authors leave a more systematic study of AIME to future work.
The optimal assumed noise level p is model-dependent, best at p=0.15 for the 1.5B model and p=0.10 for the 7B model, and the authors explicitly do not treat the optimal p as an estimate of the true mismatch rate. On MATH500 Levels 4 and 5, the differences between S-GRPO and GRPO are not statistically significant and confidence intervals remain wide, so equivalence is not established. On AIME, Avg@32 is only about 0% to 14%, and method differences correspond to a few correct samples out of 960, which is hard to interpret.