Verifiable-reward RL gets its first prompt-level privacy guarantee, and the reward signal still lifts accuracy through the noise
Synopsis
Reinforcement learning with verifiable rewards (RLVR) trains a language model on problems that may themselves be confidential, and the released weights can reveal which problems it saw. DP-GRPO aggregates all responses to one prompt into a single gradient, clips it once and adds Gaussian noise, so the privacy budget depends on neither the number of responses per prompt nor the clipping norm; the authors describe it as the first differential privacy guarantee for RLVR training. Training Qwen2.5-1.5B-Instruct with LoRA at a per-run budget of 8, the reward signal beats a control that removes only the reward by 2.65 points on MATH and 3.24 on GSM8K, positive in every seed.
Interpretation
Under prompt-level differential privacy, verifier rewards remain a usable learning signal: within one private pipeline, DP-GRPO, which keeps the group-relative advantages, is more accurate than a control that zeroes every advantage. No differential privacy guarantee existed for RLVR or GRPO training, so the value of the reward signal under privacy had not been measured; the control arm keeps generation, verification, the KL term, clipping, noise and optimizer updates and removes only the reward preference. On the same prompts and at the same per-run budget, DP-GRPO exceeds the zero-advantage control by 2.65 points on MATH and 3.24 on GSM8K, positive in all four seeds, while the control stays within 0.4 points of the base.
At a matched privacy budget, reward optimization is the better way to spend it than private supervised fine-tuning, at least on the two mathematical-reasoning benchmarks. The two private imitation routes target verifier-filtered self-samples and the dataset's reference solutions respectively, under the same sampling, clipping, noise and accounting. DP-GRPO exceeds verifier-filtered and gold DP-SFT by 2.32 to 3.79 points on MATH and GSM8K, all eight seed-paired contrasts significant after Holm adjustment, while both supervised arms stay within their minimum detectable effects of the base.
On these mathematical tasks the price of privacy is small, and the budget can be tightened without a measured accuracy loss. Against non-private GRPO measured over its own base, the private run keeps most of the gain; earlier evaluations of private post-training did not isolate the reward signal. DP-GRPO retains 85% of the non-private gain on MATH and 90% on GSM8K; on MATH an eightfold tighter budget, moving the noise multiplier from 0.484 to 1.002, costs at most 1.2 points, and halving or doubling the clipping norm changes accuracy by less than its minimum detectable effect.
Perspective
The results come from a single model, Qwen2.5-1.5B-Instruct with rank-4 LoRA, and the main evidence is two mathematical-reasoning benchmarks whose verifiers check a final answer; CommonsenseQA only tests whether the reward effect carries beyond mathematics. The budget, clipping-norm and rollout-allocation sweeps ran on MATH alone, and transfer was checked once by scoring the GSM8K checkpoints on SVAMP without further training.
The unit of the guarantee is a prompt, not a person: if one person contributes several prompts, that person's protection follows only through group privacy and degrades with the number of prompts. The theorem covers the analyzed mechanism under four assumptions; batched inference kernels and the verifier's wall-clock limit can couple one prompt's rollouts to other records, and floating-point noise generation departs from the real-number model, neither quantified. The privacy bound is certified rather than measured, and a canary audit returned no positive lower bound. On CommonsenseQA DP-GRPO retains only 45% of the non-private gain, and that non-private reference keeps the private hyperparameters rather than a tuned ceiling.