# Reward-Model Bias Mitigation

The work proposes BiasReducer, a lightweight framework that edits only the linear reward head without retraining the reward model: an SAE-style encoder with semantic supervision learns internal representations for predefined attributes such as length, confidence, and sycophancy; the framework then determines the direction and amount by which to adjust the reward head for each attribute and stores these as an edit bank; for a new dataset it ranks attributes by their influence on reward scores and applies the matching edits. Across five reward models, BiasReducer-M improves RM-Bench-Hard, Arena-StyleConflict, and JudgeBiasBench by 8.3, 18.0, and 6.

The work proposes BiasReducer, a lightweight framework that edits only the linear reward head without retraining the reward model: an SAE-style encoder with semantic supervision learns internal representations for predefined attributes such as length, confidence, and sycophancy; the framework then determines the direction and amount by which to adjust the reward head for each attribute and stores these as an edit bank; for a new dataset it ranks attributes by their influence on reward scores and applies the matching edits. Across five reward models, BiasReducer-M improves RM-Bench-Hard, Arena-StyleConflict, and JudgeBiasBench by 8.3, 18.0, and 6.