Offline RLVR stops supervising every token: GIST splits data by outcome and beats seven selectors on a shared pool
Synopsis
Offline reinforcement learning with verifiable rewards usually supervises every token of every response in a fixed sampled pool, although responses and tokens are not equally informative. GIST splits the bidirectional KL mismatch between the policy and the Gibbs-optimal target of KL-regularized RLVR into a forward and a reverse term, then pairs correct and incorrect responses by sampling temperature and supervises only low-probability tokens of correct responses and high-probability tokens of incorrect ones. On Qwen3-0.6B/GSM8K with all methods choosing from the same candidates, it supervises 66.6% fewer tokens than training on all candidates and reaches 45.79 strict and 49.73 flexible accuracy, above seven selection baselines.
Interpretation
Data selection for offline RLVR moves from a single level to two linked levels, response and token: correct and incorrect responses carry different mismatch directions, so they need different sampling sources and different supervised positions. Rejection sampling kept only verified-correct responses, reward-weighted methods such as BOLT weighted responses by reward, and other methods picked tokens by entropy or probability; each selected at one level and left open how to combine response selection with outcome-dependent token supervision. On Qwen3-0.6B/GSM8K, with every method drawing on the same candidate pool, the same eligible prompts and the same BOLT objective, GIST reaches 45.79 strict and 49.73 flexible accuracy, 1.59 to 3.33 points above all seven baselines on strict and 1.13 to 3.26 points on flexible.
The selection rule uses only three quantities available while data are being constructed: the sampling temperature, the verifier's correct-or-incorrect outcome, and the initial policy's token probabilities. The target probabilities and soft values in the analysis are not available at data-construction time, so the method approximates them from observable signals instead of estimating them. Each prompt generates responses at temperature 1 and temperature 0 without top-p or top-k truncation; correct responses come from the on-policy pool and incorrect ones from the near-mode pool, a prompt is eligible with at least 4 correct and 4 incorrect responses, and 4 of each are sampled; the token mask thresholds at the response's mean probability, is computed once with the frozen initial policy and stays fixed during training.
Perspective
The demonstrated scope is three models (Qwen3-0.6B, Qwen3.5-2B, Gemma-4-E2B-it), two offline objectives (BOLT and SFT) and three benchmarks (GSM8K, MATH-500, MBPP-full), with selection done once before training; the authors suggest the view may extend to other offline objectives and to settings where data are re-selected as the policy changes.
The budgeted-selection guarantee concerns only the mismatch captured at a fixed policy, not the outcome of training on the selected set; the quotas are a practical choice rather than a consequence of the analysis, and the cost is that the hardest prompts are excluded; masked positions still enter the forward and backward passes, so the compute saving over training on all candidates comes from keeping about a quarter of the responses; gains on MATH-500 are smaller, and Qwen3.5-2B trails BOLT by 0.60 points on MBPP-full.