# Reinforcement Learning with Verifiable Rewards

At the low learning rates common in reinforcement learning with verifiable rewards, mixed-precision BF16 training converges more slowly than FP16 because small updates to FP32 shadow weights are often erased when those weights are rounded to BF16, leaving the weights unchanged for several steps. Replacing round-to-nearest with stochastic rounding in that quantization step makes BF16 follow FP16's reward trajectory on 1.5B, 14B and 30B-A3B models across two math datasets, while drifting less from the reference model than BF16 at a raised learning rate and keeping BF16's dynamic range without loss scaling.

At the low learning rates common in reinforcement learning with verifiable rewards, mixed-precision BF16 training converges more slowly than FP16 because small updates to FP32 shadow weights are often erased when those weights are rounded to BF16, leaving the weights unchanged for several steps. Replacing round-to-nearest with stochastic rounding in that quantization step makes BF16 follow FP16's reward trajectory on 1.5B, 14B and 30B-A3B models across two math datasets, while drifting less from the reference model than BF16 at a raised learning rate and keeping BF16's dynamic range without loss scaling.

Reinforcement learning with verifiable rewards (RLVR) trains a language model on problems that may themselves be confidential, and the released weights can reveal which problems it saw. DP-GRPO aggregates all responses to one prompt into a single gradient, clips it once and adds Gaussian noise, so the privacy budget depends on neither the number of responses per prompt nor the clipping norm; the authors describe it as the first differential privacy guarantee for RLVR training. Training Qwen2.5-1.5B-Instruct with LoRA at a per-run budget of 8, the reward signal beats a control that removes only the reward by 2.65 points on MATH and 3.24 on GSM8K, positive in every seed.

Reinforcement learning with verifiable rewards (RLVR) trains a language model on problems that may themselves be confidential, and the released weights can reveal which problems it saw. DP-GRPO aggregates all responses to one prompt into a single gradient, clips it once and adds Gaussian noise, so the privacy budget depends on neither the number of responses per prompt nor the clipping norm; the authors describe it as the first differential privacy guarantee for RLVR training. Training Qwen2.5-1.5B-Instruct with LoRA at a per-run budget of 8, the reward signal beats a control that removes only the reward by 2.65 points on MATH and 3.24 on GSM8K, positive in every seed.

Long-context RLVR rollout generation now drafts tokens with sparse attention and verifies them with full attention, raising end-to-end training throughput to 1.39x and 1.35x of autoregressive on Qwen3-1.7B and Qwen3-8B while keeping comparable task performance.

Long-context RLVR rollout generation now drafts tokens with sparse attention and verifies them with full attention, raising end-to-end training throughput to 1.39x and 1.35x of autoregressive on Qwen3-1.7B and Qwen3-8B while keeping comparable task performance.

Sustained concurrency in asynchronous RLVR rollout concentrates context processing into fewer decode steps, making the KV cache rather than weights the dominant cost. KV4-RL stores the full context in calibration-free INT4 and protects only attention sinks and recent context in bounded INT8; across full-parameter GRPO training on Qwen3-4B-Base and Qwen3-8B-Base, its six-benchmark mean accuracy stays within 0.5 percentage points of BF16 rollout while substantially accelerating rollout and full iterations.

Sustained concurrency in asynchronous RLVR rollout concentrates context processing into fewer decode steps, making the KV cache rather than weights the dominant cost. KV4-RL stores the full context in calibration-free INT4 and protects only attention sinks and recent context in bounded INT8; across full-parameter GRPO training on Qwen3-4B-Base and Qwen3-8B-Base, its six-benchmark mean accuracy stays within 0.5 percentage points of BF16 rollout while substantially accelerating rollout and full iterations.

Distillation no longer treats every teacher token alike: each token's weight is set by its predicted effect on the student's post-update reward. Earlier token-selection methods relied on proxies such as uncertainty or teacher-student divergence, which need not identify reward-improving supervision. RAD scores each token by differentiating through a temporary optimizer step, then updates token weights by projected ascent, improving mean accuracy on six competition-math benchmarks by 6.5 points for Qwen3-1.7B and 2.5 points for Qwen3-4B-Instruct-2507 at a matched training-response budget, and reaching 54.69% and 58.67% pass@1 on LiveCodeBench-v6.

Distillation no longer treats every teacher token alike: each token's weight is set by its predicted effect on the student's post-update reward. Earlier token-selection methods relied on proxies such as uncertainty or teacher-student divergence, which need not identify reward-improving supervision. RAD scores each token by differentiating through a temporary optimizer step, then updates token weights by projected ascent, improving mean accuracy on six competition-math benchmarks by 6.5 points for Qwen3-1.7B and 2.5 points for Qwen3-4B-Instruct-2507 at a matched training-response budget, and reaching 54.69% and 58.67% pass@1 on LiveCodeBench-v6.

The work proposes RL-ARC, a calibration-aware reinforcement learning framework that, on top of RLVR, elicits both the model's confidence in its reasoning process and its confidence in its answer, using reasoning confidence as an auxiliary signal: reasoning-guided regularization when the prediction is correct and an overconfidence penalty when it is incorrect; across mathematical reasoning and out-of-distribution complex reasoning and factual QA benchmarks, RL-ARC improves calibration metrics over existing calibration-aware training methods while keeping accuracy comparable to RLVR, which optimizes correctness alone.

The work proposes RL-ARC, a calibration-aware reinforcement learning framework that, on top of RLVR, elicits both the model's confidence in its reasoning process and its confidence in its answer, using reasoning confidence as an auxiliary signal: reasoning-guided regularization when the prediction is correct and an overconfidence penalty when it is incorrect; across mathematical reasoning and out-of-distribution complex reasoning and factual QA benchmarks, RL-ARC improves calibration metrics over existing calibration-aware training methods while keeping accuracy comparable to RLVR, which optimizes correctness alone.

The study builds corpus-grounded QA data with intermediate evidence annotations and compares three retrieval signals (evidence coverage Cov, coverage with dependencies Cov-Dep, answer match AM) under scalar reward bonuses versus action-aligned local credit, training Qwen3-4B on seven QA benchmarks and finding local credit outperforms scalar, with Cov-local reaching 54.34 average F1 versus 51.25 for outcome-only training.

The study builds corpus-grounded QA data with intermediate evidence annotations and compares three retrieval signals (evidence coverage Cov, coverage with dependencies Cov-Dep, answer match AM) under scalar reward bonuses versus action-aligned local credit, training Qwen3-4B on seven QA benchmarks and finding local credit outperforms scalar, with Cov-local reaching 54.34 average F1 versus 51.25 for outcome-only training.

The work proposes a "visual resolvability" design rule requiring that correct answers depend on visual evidence while the task stays learnable, and tests it on counterfactual coordinate scenes with standard GRPO and correctness-and-format rewards on Qwen2.5-VL-7B: on held-out 20-point scenes denser than any seen in training, discovery accuracy rises from 0.425 to 0.875; replacing test images with gray canvases drops accuracy to zero, while training on gray canvases at matched step 30 across four seeds yields essentially none of the gain, indicating the gain comes from training images; adding a caption that answers the training question, with the same images, reward and budget, cuts the gain by nearly two thirds.

The work proposes a "visual resolvability" design rule requiring that correct answers depend on visual evidence while the task stays learnable, and tests it on counterfactual coordinate scenes with standard GRPO and correctness-and-format rewards on Qwen2.5-VL-7B: on held-out 20-point scenes denser than any seen in training, discovery accuracy rises from 0.425 to 0.875; replacing test images with gray canvases drops accuracy to zero, while training on gray canvases at matched step 30 across four seeds yields essentially none of the gain, indicating the gain comes from training images; adding a caption that answers the training question, with the same images, reward and budget, cuts the gain by nearly two thirds.