# Multi-reward reinforcement learning

The work observes that when GRPO trains with multiple rewards it normalizes the summed reward by its within-group standard deviation, whose variance equals the sum of all pairwise reward covariances, so large-scale rewards can dominate the normalization and suppress smaller-scale reward signals; the authors propose CorrGRPO, which replaces pairwise covariances in the denominator with Pearson correlation coefficients while keeping the centered total reward unchanged, and compare it with GRPO and variants such as GDPO on code generation, tool calling, and agent utility versus security across models from 0.5B to 8B parameters, reporting improvements in all three domains and an outward expansion of the empirical Pareto frontier between competing objectives.

The work observes that when GRPO trains with multiple rewards it normalizes the summed reward by its within-group standard deviation, whose variance equals the sum of all pairwise reward covariances, so large-scale rewards can dominate the normalization and suppress smaller-scale reward signals; the authors propose CorrGRPO, which replaces pairwise covariances in the denominator with Pearson correlation coefficients while keeping the centered total reward unchanged, and compare it with GRPO and variants such as GDPO on code generation, tool calling, and agent utility versus security across models from 0.5B to 8B parameters, reporting improvements in all three domains and an outward expansion of the empirical Pareto frontier between competing objectives.

The work characterizes each reward's contribution to policy updates in multi-reward RL through "advantage energy," shows that under idealized GDPO normalization a reward's advantage energy is proportional to its active-group density (the fraction of rollout groups in which it yields nonzero relative advantages), and proposes DARA, an inverse-square-root density weighting that reaches target behaviors faster than GDPO: up to 26% fewer training steps for format compliance in tool calling and up to 65% fewer steps for near-saturated length compliance in mathematical reasoning, while remaining competitive in final performance.

The work characterizes each reward's contribution to policy updates in multi-reward RL through "advantage energy," shows that under idealized GDPO normalization a reward's advantage energy is proportional to its active-group density (the fraction of rollout groups in which it yields nonzero relative advantages), and proposes DARA, an inverse-square-root density weighting that reaches target behaviors faster than GDPO: up to 26% fewer training steps for format compliance in tool calling and up to 65% fewer steps for near-saturated length compliance in mathematical reasoning, while remaining competitive in final performance.