When every rollout fails, PICO turns all-wrong groups into trainable trajectories with teacher-constrained joint decoding
Synopsis
Verifiable-reward reinforcement learning hits a learning cliff on problems beyond the model's reach: when a rollout group is entirely wrong, group-relative advantage collapses to zero and the update carries no signal. PICO triggers distribution-constrained joint decoding only on all-failure groups, intersecting the teacher's next-token candidates with the student's while a dynamic cutoff keeps the sampling distribution inside a finite divergence budget, yielding trajectories that pass verification and stay close to the current policy. On Qwen3-4B-Base and Qwen3-8B-Base, PICO averages 32.0 and 35.4 across AIME25, AIME26, HMMT26, MATH500, GPQA-Diamond and HLE, against 27.9 and 31.8 for GRPO.
Interpretation
When every sample on a hard problem fails, training no longer gets only an empty zero-advantage update; it recovers verified positive-reward trajectories from unsolved problems. Group-relative methods such as GRPO collapse the advantage to zero in an all-wrong group, so those problems supply no learning signal, while fitting expert trajectories by supervised fine-tuning or mixing them into rollout groups causes policy drift or distribution mismatch because they sit in the student's low-probability region. In mathematical-reasoning training, joint decoding recovers verified positive-reward trajectories in up to 80% of all-failure groups; on training curves PICO holds higher reward and fewer solve-none questions, and while baselines plateau after the initial stage PICO keeps improving.
The mechanism intersects the teacher's next-token candidates with the student's at the token level, then samples according to the student's own relative probabilities. The teacher sees the expert solution and the student does not; expert information enters generation only through the teacher's next-token predictions and is never written into the student's context. The student cutoff is fixed at 50 and the teacher cutoff is capped at 20, taking the smallest teacher cutoff whose intersection retains at least 0.8 of the student probability mass, with a fallback to the student policy when no cutoff is feasible; during training the retained student probability mass stays consistently high and the token-level KL divergence falls accordingly.
Rescued trajectories are optimized together with on-policy rollouts under a mixed-behavior objective that uses an importance-sampling ratio to correct the behavior-policy difference. Trajectories from joint decoding come from a behavior policy different from the student, so treating them as on-policy samples would introduce bias. In ablations, removing joint decoding drops the average from 37.2 to 33.8 and removing the off-policy importance-sampling ratio drops it to 35.3, while dynamic top-k at 37.2 exceeds each fixed-cutoff setting.
Perspective
The demonstrated scope is three student policies, Qwen3-4B-Base, Qwen3-8B-Base and Qwen3-4B-Instruct, with Qwen3-4B-Instruct conditioned on expert solutions as the teacher throughout; math and general reasoning train on DAPO-Math-17K, agentic tasks train separately on ALFWorld and ScienceWorld, and at evaluation the student works independently without the teacher or joint decoding.
Joint decoding carries a clear wall-clock cost, raising time per training step from 28.3 to 43.6 minutes while utilization per minute stays comparable to GRPO; the importance-sampling ratio is exact per token only on sampled actions, and the hard support restriction drops contributions from tokens outside the support, so the mixed-behavior objective is a surrogate rather than an unbiased estimator of the original on-policy objective.