# Density-Aware Reward Aggregation

The work characterizes each reward's contribution to policy updates in multi-reward RL through "advantage energy," shows that under idealized GDPO normalization a reward's advantage energy is proportional to its active-group density (the fraction of rollout groups in which it yields nonzero relative advantages), and proposes DARA, an inverse-square-root density weighting that reaches target behaviors faster than GDPO: up to 26% fewer training steps for format compliance in tool calling and up to 65% fewer steps for near-saturated length compliance in mathematical reasoning, while remaining competitive in final performance.

The work characterizes each reward's contribution to policy updates in multi-reward RL through "advantage energy," shows that under idealized GDPO normalization a reward's advantage energy is proportional to its active-group density (the fraction of rollout groups in which it yields nonzero relative advantages), and proposes DARA, an inverse-square-root density weighting that reaches target behaviors faster than GDPO: up to 26% fewer training steps for format compliance in tool calling and up to 65% fewer steps for near-saturated length compliance in mathematical reasoning, while remaining competitive in final performance.