Discounting rewards by density: DARS stops GRPO collapsing onto one high-reward mode, cutting mode-mass error to 0.014 on a four-peak navigation task
Synopsis
Reinforcement learning that maximizes expected return tends to concentrate probability on a few high-reward outcomes; DARS estimates the local density of outcomes within each on-policy rollout group and inversely scales rewards by that density before standard GRPO updates. On a four-Gaussian navigation task, GRPO covers a single mode while DARS-GRPO recovers mode masses close to 0.25 each, with mode total variation 0.014 versus 0.750 for GRPO. In text-to-image prompt adaptation, DARS-GRPO raises image and prompt diversity across datasets, policy language models and reward settings while keeping semantic alignment.
Interpretation
One extra step in reward processing: among outcomes sampled together, rewards in dense regions are scaled down and high-reward outcomes in sparse regions gain relative weight, while the policy-gradient objective itself is unchanged. Earlier inverse probability scaling divided rewards by how often an outcome was revisited, but in continuous outcome spaces exact repeats are rare, so that frequency estimate does not carry over directly. On a four-Gaussian reward navigation task over the unit square, GRPO puts all mass on one mode, while DARS-GRPO reaches mode total variation 0.014 and minimum mode mass 0.236, against 0.750 and 0.000 for GRPO.
The method computes local density in a feature representation of outcomes, summing Gaussian-kernel similarities between each outcome and the rest of its group to get a density score, then divides the reward by that score. Density is estimated only within one on-policy rollout group, requiring no learned density model, critic or auxiliary loss, and no change to the policy architecture. The text-to-image prompt adaptation experiments map each generated image to a 768-dimensional feature with a frozen DINOv2 encoder, use kernel width 20, train on 1,024 MS-COCO prompts and evaluate on held-out MS-COCO and Lexica prompts.
The kernel width has to sit in an intermediate range for density estimation to resolve meaningful neighborhoods. With a very small width nearly every image looks isolated and density is dominated by self-similarity, making reward scaling almost uniform; with an excessively large width distant images count as neighbors and density is oversmoothed. A kernel-width ablation sweeps widths 10, 15, 20, 25 and 30, with diversity metrics such as Image Vendi peaking at intermediate widths while image quality stays largely unchanged.
Perspective
The demonstrated scope is a four-Gaussian reward navigation task with a known target distribution, plus text-to-image prompt adaptation that trains only the prompt-rewriting policy against a frozen Stable Diffusion v1.4 generator, evaluated on MS-COCO and Lexica prompts with Qwen3-0.6B and GPT-2 as policy language models under two reward settings.
The results are an empirical approximation to reward-proportional sampling rather than a general convergence guarantee; the method relies on a representation space whose distances reflect semantic similarity and on a suitable kernel width, and may degrade when embedding distances poorly reflect similarity or no single bandwidth captures the relevant neighborhood structure; pairwise density computation also grows quadratically with rollout-group size, potentially limiting use with large groups.