Kuaishou team's DARA weights rewards by active-group density, cutting tool-calling format steps by up to 26% and math length-compliance steps by up to 65%
Synopsis
The work characterizes each reward's contribution to policy updates in multi-reward RL through "advantage energy," shows that under idealized GDPO normalization a reward's advantage energy is proportional to its active-group density (the fraction of rollout groups in which it yields nonzero relative advantages), and proposes DARA, an inverse-square-root density weighting that reaches target behaviors faster than GDPO: up to 26% fewer training steps for format compliance in tool calling and up to 65% fewer steps for near-saturated length compliance in mathematical reasoning, while remaining competitive in final performance.
Interpretation
The paper introduces advantage energy, the sum of a reward's squared advantages over a batch, as a measure of how much signal each reward supplies to the policy update, and shows that under idealized GDPO normalization every active group contributes equal energy, so the batch total is proportional to active-group density. GDPO preserves reward-specific relative information through reward-wise normalization, but the paper identifies a residual batch-level imbalance: total energy scales linearly with active-group density, so sparse rewards are drowned out. The relation is derived (Equation 5 and Appendix A.1) and supported on training logs: at step 40 in the mathematical reasoning experiments the length reward is active in fewer groups than correctness, and under GDPO its energy is 31–46% of correctness.
From this relation the paper derives an inverse-square-root density correction and builds DARA, which gives larger weights to less frequently active rewards relative to the highest-density reward in the batch, recomputing weights per rollout batch without modifying the underlying policy optimization objective. Unlike DVAO (within-group standard deviation), SAW (batch-level coefficient of variation), SA-MRPO (saturation) and GD2PO (sign consistency of advantages), DARA's weights follow directly from the density–energy relationship. Two implementations are given: DARA-Sym scales both signs and can match energy exactly, while DARA-Asym amplifies only positive advantages and comes with energy bounds (Appendix A.5); DARA-Asym is the main method.
On tool calling, DARA acquires the format objective faster than GDPO: the median format reward reaches 0.8 at steps 14–15 versus 19 for GDPO and 34 for GRPO, and at step 60 DARA-Sym and DARA-Asym achieve the highest Average Accuracy (50.94% and 50.69%) and Average Format (96.06% and 96.02%), against at most 50.50% and 90.11% for baselines. The paper attributes the speedup to density calibration compensating for sparse reward activity and offers a testable prediction—at a fixed budget of 2,048 responses, larger groups raise active-group density and accelerate learning—which the experiments match. Tool-calling experiments use ToolRL's 3,920 training and 80 held-out examples on Qwen2.5-1.5B-Instruct and 3B-Instruct, comparing GRPO, GDPO, DVAO and GD2PO-Hard, with 5 training seeds for the main runs; at the step-100 final checkpoints DARA is competitive rather than uniformly ahead.
On mathematical reasoning, DARA's advantage grows as the length reward approaches saturation: on Qwen3-4B-Instruct, DARA-Asym, DARA-Sym and GDPO reach 99% length compliance at steps 66, 59 and 111, and on DeepSeek-R1-7B at steps 56, 50 and 142, corresponding to 41–65% fewer steps than GDPO; on DeepSeek-R1-7B both DARA variants first reach 95% held-out length compliance at step 50 versus step 70 for GDPO. The paper explains that early on many groups mix compliant and over-length responses so the length reward is informative, while near saturation such mixed groups become rare—exactly the regime where density calibration acts; a fixed-weight ablation shows that simply raising the length weight improves compliance but costs more accuracy. Math experiments cover DeepSeek-R1-1.5B, Qwen3-4B-Instruct and DeepSeek-R1-7B on DeepScaleR-Preview data and five public benchmarks (1,559 questions, 24,944 generations per checkpoint), with 2,000-resample confidence intervals.
Perspective
The result targets teams doing LLM post-training with GRPO/GDPO-style multi-reward RL, in settings where rewards decompose into channels whose activity frequencies differ markedly, such as format versus correctness in tool calling or correctness versus length in mathematical reasoning. The method replaces only the reward aggregation step and can be layered onto existing training stacks; the paper reports that at a fixed budget of 2,048 responses, larger groups raise active-group density and accelerate learning, which informs batch and group-size configuration. DARA-Asym and DARA-Sym show different accuracy–compliance trade-offs: the former attains the highest Joint score on both DeepSeek-R1 scales, the latter the lowest exceed rates, so readers can pick a variant by goal.
The exact energy-matching result rests on idealized GDPO normalization and an explicit covariance condition, while the deployed DARA-Asym amplifies only positive advantages and has energy bounds rather than exact matching, leaving a gap between the theory and the main method that readers must judge for themselves. Speedups are measured at specific thresholds (99% length compliance, median format reward 0.8) and specific model scales; whether the magnitudes hold under other reward definitions, thresholds or backbones is not settled in the text. At the step-100 final checkpoints the gap to baselines narrows considerably, suggesting the value lies mainly in optimization dynamics rather than a higher final ceiling. The reading covered the full text, but parts of the derivations and comparison tables sit in the appendices, so reproducing the weight cap and the exact advantage estimators still requires Appendices A, B, C and D.