Skip to main content

# Reinforcement Learning from Human Feedback

2 related articles

Subscribe to updates

arXiv Nov 2, 2026
Before spending GPU-hours on RL, score two reward models: within-prompt covariance explained 82% of the response to controlled edits
Treating an edit to a reward model as a price change on response features, a team forecasts before training which features a policy will supply more or less of, using samples from the current policy and the score difference between the old and new reward models. Across 54 RL runs from six parent policies, the covariance predictor explained 82% of the standardized response at 200 steps, against 50% for the classical counterfactual-price alternative; the same forecast explained 66% out of sample on three real reward-model upgrades, but its ranking of edits by rent held only over the first 30 steps.
Treating an edit to a reward model as a price change on response features, a team forecasts before training which features a policy will supply more or less of, using samples from the current policy and the score difference between the old and new reward models. Across 54 RL runs from six parent policies, the covariance predictor explained 82% of the standardized response at 200 steps, against 50% for the classical counterfactual-price alternative; the same forecast explained 66% out of sample on three real reward-model upgrades, but its ranking of edits by rent held only over the first 30 steps.
Zixiao Wang（First author）
Forecasting Reward-Model Edits Reinforcement Learning from Human Feedback Reward Model Reward Model Overoptimization
arXiv Sep 29, 2026
DRM replaces the scalar reward head with a diffusion head: matching or beating same-data, same-backbone baselines on five benchmarks while its reward distribution turns multimodal as human disagreement grows
The work introduces DRM (Diffusion Reward Model), which recasts reward modeling as conditional density estimation: on top of a frozen LLM encoder, a lightweight Diffusion Transformer denoises Gaussian noise into a reward vector with no parametric family assumed, a single architecture handles both multi-attribute regression and pairwise preference data, it matches or surpasses the same-data, same-backbone scalar head ArmoRM and quantile head QRM across five benchmarks, and it exhibits multimodal structure tied to human disagreement on repeated-annotation data, with uncertainty-aware rejection, LCB aggregation, and downstream RLHF experiments demonstrating the practical value of distributional information.
The work introduces DRM (Diffusion Reward Model), which recasts reward modeling as conditional density estimation: on top of a frozen LLM encoder, a lightweight Diffusion Transformer denoises Gaussian noise into a reward vector with no parametric family assumed, a single architecture handles both multi-attribute regression and pairwise preference data, it matches or surpasses the same-data, same-backbone scalar head ArmoRM and quantile head QRM across five benchmarks, and it exhibits multimodal structure tied to human disagreement on repeated-annotation data, with uncertainty-aware rejection, LCB aggregation, and downstream RLHF experiments demonstrating the practical value of distributional information.
Tsinghua University University of Illinois Urbana-Champaign
Xiang Wang（First author）
Diffusion Reward Model (DRM)Reward Modeling Diffusion Transformer Reinforcement Learning from Human Feedback Test-Time Scaling

You’re all caught up