# Forecasting Reward-Model Edits

Treating an edit to a reward model as a price change on response features, a team forecasts before training which features a policy will supply more or less of, using samples from the current policy and the score difference between the old and new reward models. Across 54 RL runs from six parent policies, the covariance predictor explained 82% of the standardized response at 200 steps, against 50% for the classical counterfactual-price alternative; the same forecast explained 66% out of sample on three real reward-model upgrades, but its ranking of edits by rent held only over the first 30 steps.

Treating an edit to a reward model as a price change on response features, a team forecasts before training which features a policy will supply more or less of, using samples from the current policy and the score difference between the old and new reward models. Across 54 RL runs from six parent policies, the covariance predictor explained 82% of the standardized response at 200 steps, against 50% for the classical counterfactual-price alternative; the same forecast explained 66% out of sample on three real reward-model upgrades, but its ranking of edits by rent held only over the first 30 steps.