# Reward Model Overoptimization

Treating an edit to a reward model as a price change on response features, a team forecasts before training which features a policy will supply more or less of, using samples from the current policy and the score difference between the old and new reward models. Across 54 RL runs from six parent policies, the covariance predictor explained 82% of the standardized response at 200 steps, against 50% for the classical counterfactual-price alternative; the same forecast explained 66% out of sample on three real reward-model upgrades, but its ranking of edits by rent held only over the first 30 steps.

Treating an edit to a reward model as a price change on response features, a team forecasts before training which features a policy will supply more or less of, using samples from the current policy and the score difference between the old and new reward models. Across 54 RL runs from six parent policies, the covariance predictor explained 82% of the standardized response at 200 steps, against 50% for the classical counterfactual-price alternative; the same forecast explained 66% out of sample on three real reward-model upgrades, but its ranking of edits by rent held only over the first 30 steps.

Language models are often trained against a learned judge instead of the truth, but what that judge rewards could previously be seen only after training, from the policy's behavior. Three judges were trained from a 1B-parameter model on 500, 2,000 and 8,000 labeled answers, and a single-edit probe measured each judge's lift on wrong answers before any training. The probe predicted a climb onto flagged text against the weakest judge; after training, 115 of 128 final responses were one Sahara sentence ending "Answer: Africa", and the judge's mean score rose from 0.20 to 0.82. Against the strongest judge it predicted nothing climbable, and that run was not exploited.

Language models are often trained against a learned judge instead of the truth, but what that judge rewards could previously be seen only after training, from the policy's behavior. Three judges were trained from a 1B-parameter model on 500, 2,000 and 8,000 labeled answers, and a single-edit probe measured each judge's lift on wrong answers before any training. The probe predicted a climb onto flagged text against the weakest judge; after training, 115 of 128 final responses were one Sahara sentence ending "Answer: Africa", and the judge's mean score rose from 0.20 to 0.82. Against the strongest judge it predicted nothing climbable, and that run was not exploited.