Before spending GPU-hours on RL, score two reward models: within-prompt covariance explained 82% of the response to controlled edits
Synopsis
Treating an edit to a reward model as a price change on response features, a team forecasts before training which features a policy will supply more or less of, using samples from the current policy and the score difference between the old and new reward models. Across 54 RL runs from six parent policies, the covariance predictor explained 82% of the standardized response at 200 steps, against 50% for the classical counterfactual-price alternative; the same forecast explained 66% out of sample on three real reward-model upgrades, but its ranking of edits by rent held only over the first 30 steps.
Interpretation
The effect of an edit to a reward model can be computed from samples of the existing policy before reinforcement learning starts, instead of waiting for a full training run. Teams that edit reward models, by adding penalties, retraining on cleaner data, ensembling or auditing a fraction of samples, previously learned only after an RL run whether rent fell and what the policy produced instead, and a run costs hundreds of GPU-hours, so most edits were evaluated on short runs or not at all. Across 54 RL runs from six parent policies, with every forecast fixed before the first run, the covariance predictor predicted the direction of all 20 supply changes whose forecast had the same sign at every parent and explained 82% of the standardized response at the 200-step window; the counterfactual-price predictor explained 50%, with a bootstrap interval for the difference of 0.29 to 0.39.
The forecast needs only samples from the current policy and scores from the two reward models, not a list of which features carry prices. The price predictor requires the reward change to reduce to a linear repricing of measured features, and that reduction fails once an edit changes reward outside their span. The gap sits in length: on the two formatting goods the predictors explain similar shares (0.71 and 0.87 for headers, 0.91 and 0.66 for list markers), while on length the price predictor explains 7% and the covariance predictor 84%; on three real reward-model upgrades the covariance predictor with frozen coefficients predicted all 7 same-sign directions and explained 66%, and under GRPO it predicted all 22 stable changes and explained 75%.
The same covariance ranks reward-model edits by their effect on rent within the first 30 steps, and that ranking has a measured horizon. Rent is the gap between proxy reward and a gold reward from stronger judges, and previously there was no pre-training way to tell whether an edit would raise or lower it. Over steps 20 to 30 the forecast agreed with the ranking of the six arms by change in rent (Spearman 0.94) and by change in gold reward (0.89); by step 200 the ranking of the factorial arms reversed under both RLOO and GRPO, while the three real upgrades kept their order, and none of those upgrades lowered the price of headers.
Perspective
The results come from one testbed: a 1.7B Qwen3 policy and one family of reward models as proxies, trained mostly with RLOO and in one batch with GRPO, with first-order forecasts tested over 200 steps after a branch point. The instrument applies to features that can be detected by rules and deleted cleanly, such as headers and bold, list markers, a closing summary and a resource list; the rent forecast holds only in the early phase of training.
The ranking of rent reverses by step 200, and the analyses of that reversal are exploratory; in a prospective test with symmetric shocks the policy moved further after a price cut, but gold reward did not fall with that distance, and the effective coefficient describes price rises but not price cuts. Two arms with a large shock to summaries ran away in length, and one of 16 implants fell just below the validation threshold on parent responses.