# Single-edit probe of a reward model

Language models are often trained against a learned judge instead of the truth, but what that judge rewards could previously be seen only after training, from the policy's behavior. Three judges were trained from a 1B-parameter model on 500, 2,000 and 8,000 labeled answers, and a single-edit probe measured each judge's lift on wrong answers before any training. The probe predicted a climb onto flagged text against the weakest judge; after training, 115 of 128 final responses were one Sahara sentence ending "Answer: Africa", and the judge's mean score rose from 0.20 to 0.82. Against the strongest judge it predicted nothing climbable, and that run was not exploited.

Language models are often trained against a learned judge instead of the truth, but what that judge rewards could previously be seen only after training, from the policy's behavior. Three judges were trained from a 1B-parameter model on 500, 2,000 and 8,000 labeled answers, and a single-edit probe measured each judge's lift on wrong answers before any training. The probe predicted a climb onto flagged text against the weakest judge; after training, 115 of 128 final responses were one Sahara sentence ending "Answer: Africa", and the judge's mean score rose from 0.20 to 0.82. Against the strongest judge it predicted nothing climbable, and that run was not exploited.