GRPO's group baseline can now tilt two ways: strategy-based subgroup offsets lift multi-sample coverage modestly but consistently
Synopsis
GRPO scores every rollout of a prompt against one shared group-mean baseline, yet rollouts of the same prompt often follow different solution strategies with different reward distributions. This work defines the subgroup offset as a subgroup mean's deviation from the global mean and shifts the baseline along it by a single signed coefficient: the positive sign pulls credit toward a rollout's standing inside its subgroup, the negative sign pulls it toward the subgroup's overall quality. On Qwen3-8B-Base and Olmo3-7B-Base across four competition-level math benchmarks, both directions give modest but consistent gains over an anchor that shares the same strategy-conditioned sampling and differs only in the baseline, concentrated in pass@16, with the negative direction stronger.
Interpretation
Rollouts of one prompt are no longer treated as a homogeneous group; the baseline can move separately for each solution-strategy subgroup they belong to. Standard GRPO uses the within-group mean reward as a shared baseline, so equally rewarded rollouts receive equal advantage, reflecting neither a rollout's position inside its subgroup nor that subgroup's position among the rest. The comparison is Strategy-GRPO, which shares the same strategy-conditioned sampling and differs only in whether the baseline is adjusted, so the difference can be attributed to the baseline.
The method first has the model write a short solution preview, partitions rollouts by that preview, and then changes only the baseline. The subgroup offset is a subgroup's mean reward minus the global mean, and the baseline moves along it by a signed coefficient while the reward signal and optimization objective stay unchanged. Training uses DAPO-Math-17K on Qwen3-8B-Base and Olmo3-7B-Base, and evaluation covers AIME24, AIME25, AIME26 and HMMT25 in a single stage without strategy conditioning.
The gains land mainly on multi-sample coverage rather than single-sample accuracy. Against the Strategy-GRPO anchor, both directions raise pass@16 modestly but consistently, with smaller improvements in pass@1. The negative direction is stronger than the positive one on both models, and randomly permuting subgroup offsets across subgroups drops both directions below the baseline.
Perspective
The demonstrated scope is two base models, Qwen3-8B-Base and Olmo3-7B-Base, trained on DAPO-Math-17K and evaluated in a single stage on four competition-level math benchmarks: AIME24, AIME25, AIME26 and HMMT25; subgroups are induced by presampled solution strategies, and the strategy itself is neither rewarded nor fed back into the policy.
The gains over the anchor are described as modest; the coefficient sweep was run only on Qwen3-8B-Base, so the finding that a moderate coefficient works best is scoped to that model; the offset-shuffling ablation shows the gains depend on the genuine pairing of subgroups and offsets but does not separate actively wrong offsets from the milder possibility that any miscalibrated offset hurts; no rule is given for adaptively choosing direction or strength.