Discounting rewards by density: DARS stops GRPO collapsing onto one high-reward mode, cutting mode-mass error to 0.014 on a four-peak navigation task
Reinforcement learning that maximizes expected return tends to concentrate probability on a few high-reward outcomes; DARS estimates the local density of outcomes within each on-policy rollout group and inversely scales rewards by that density before standard GRPO updates. On a four-Gaussian navigation task, GRPO covers a single mode while DARS-GRPO recovers mode masses close to 0.25 each, with mode total variation 0.014 versus 0.750 for GRPO. In text-to-image prompt adaptation, DARS-GRPO raises image and prompt diversity across datasets, policy language models and reward settings while keeping semantic alignment.