GRPO curricula stop following a fixed schedule: probe accuracy steers difficulty sampling, with the largest gains at intermediate levels
Synopsis
In GRPO reinforcement fine-tuning, a fixed easy-to-hard curriculum can drift from the model's actual progress. ClosedLoopGaussian keeps the Gaussian sampler's form but uses periodic probe accuracy to adjust how fast training advances and how widely it rehearses, leaving the reward, architecture and optimizer unchanged. On Countdown, mean exact-match accuracy over four seeds exceeds both the fixed curriculum and vanilla GRPO, with gains concentrated at intermediate difficulty, though seed dispersion is large and the comparison is read descriptively.
Interpretation
The difficulty distribution of training prompts is no longer set by a curriculum fixed before training, but adjusted according to the model's measured pass rates at each difficulty level. The earlier easy-to-hard Gaussian curriculum fixed its pace and spread before training and then advanced regardless of whether the model learned faster, stalled, or regressed on earlier skills. On Countdown under an identical rollout budget across four seeds, ClosedLoopGaussian's mean exact-match accuracy exceeds both the fixed Gaussian curriculum and vanilla GRPO, and it beats the fixed curriculum on each of the four seeds.
The controller changes only the sampling distribution; the model, verifier, reward function and GRPO optimizer stay as they were. The approach keeps the same Gaussian easy-to-hard sampler as the fixed baseline and makes its pace and spread parameters adjustable online, so feedback is the primary design difference between the two arms. Every fixed number of optimizer steps, the unchanged Countdown verifier greedily evaluates a fixed pre-cached set of held-out examples per difficulty bucket, producing per-bucket accuracy as the feedback signal.
Feedback splits into two paths: one speeds up or slows down movement toward harder buckets based on deviation from a reference pace, the other widens rehearsal when easier buckets regress. Earlier adaptive curricula represented curriculum state through a single target difficulty or a task-selection policy, without separating how fast to advance from how broadly to rehearse. Controller traces over four seeds show two regimes: probe accuracy rises and pace accelerates on three seeds, while one seed stays low, pace returns to its reference and spread widens.
Perspective
The findings rest on Llama-3.2-3B-Instruct with LoRA, a single Countdown symbolic arithmetic task and four seeds, all within this controlled setting. The authors list broader model, task and seed coverage, plus direct comparisons with adaptive curricula such as AdaRFT, DUMP and SEC, as future work.
The authors note that across-seed dispersion is large relative to the mean gaps and that four seeds do not support a statistical-significance claim; vanilla GRPO collapses to near-floor accuracy on two seeds, so its mean should be read against that bimodal spread. Controller thresholds, gains and bounds are manually calibrated, overly strict settings can cause curriculum deadlock, and the single-seed diagnostic does not establish a causal decomposition of pace and spread.