# DP-GRPO

Reinforcement learning with verifiable rewards (RLVR) trains a language model on problems that may themselves be confidential, and the released weights can reveal which problems it saw. DP-GRPO aggregates all responses to one prompt into a single gradient, clips it once and adds Gaussian noise, so the privacy budget depends on neither the number of responses per prompt nor the clipping norm; the authors describe it as the first differential privacy guarantee for RLVR training. Training Qwen2.5-1.5B-Instruct with LoRA at a per-run budget of 8, the reward signal beats a control that removes only the reward by 2.65 points on MATH and 3.24 on GSM8K, positive in every seed.

Reinforcement learning with verifiable rewards (RLVR) trains a language model on problems that may themselves be confidential, and the released weights can reveal which problems it saw. DP-GRPO aggregates all responses to one prompt into a single gradient, clips it once and adds Gaussian noise, so the privacy budget depends on neither the number of responses per prompt nor the clipping norm; the authors describe it as the first differential privacy guarantee for RLVR training. Training Qwen2.5-1.5B-Instruct with LoRA at a per-run budget of 8, the reward signal beats a control that removes only the reward by 2.65 points on MATH and 3.24 on GSM8K, positive in every seed.