# Reward Modeling

6 related articles

Subscribe to updates

arXiv Nov 10, 2026
Different LLMs can now collaborate federatedly: FedVP pools locally trained verifiers instead of merging parameters
Federated learning usually requires institutions to share one model backbone so parameter updates can be aggregated, which keeps centers running different model families or sizes from collaborating. FedVP moves collaboration from parameter aggregation to shared verification: each client trains a verifier on its local data and candidate responses from its own frozen LLM, and the server pools these independently trained verifiers to score and select answers, never merging backbone parameters. Across five medical question-answering datasets, with centers using Qwen and LLaMA backbones from 3B to 14B parameters, one verifier pool is reused for all five backbones without retraining and raises mean accuracy by 1.46 to 5.79 percentage points over the strongest conventional federated baselines.
Federated learning usually requires institutions to share one model backbone so parameter updates can be aggregated, which keeps centers running different model families or sizes from collaborating. FedVP moves collaboration from parameter aggregation to shared verification: each client trains a verifier on its local data and candidate responses from its own frozen LLM, and the server pools these independently trained verifiers to score and select answers, never merging backbone parameters. Across five medical question-answering datasets, with centers using Qwen and LLaMA backbones from 3B to 14B parameters, one verifier pool is reused for all five backbones without retraining and raises mean accuracy by 1.46 to 5.79 percentage points over the strongest conventional federated baselines.
Yifan Li（First author）
FedVP Federated Learning Question Answering Reward Modeling
Transactions of the Association for Computational Linguistics Oct 28, 2026
Literary translation now rewards a large model's scores: reinforcement learning on source sentences alone lifts a 7B model past QwQ-32B-preview on metaphor-heavy text
Literary translation now gets its reinforcement-learning reward from a large model scoring against written criteria, instead of human preference labels or paired reference translations. Earlier DRT reasoning translation models were supervised on synthesized long thought and translation data, and reinforcement-learning routes such as R1-T1 and MT-R1 used machine translation metrics like COMET and CometKiwi as reward. Built on Qwen2.5-7B, DeepTrans-7B improves over its backbone by 16.3%, 13.8% and 9.0% on GEA5, GEA100 and GRF on MetaphorTrans, and beats sub-30B baselines on most metrics.
Literary translation now gets its reinforcement-learning reward from a large model scoring against written criteria, instead of human preference labels or paired reference translations. Earlier DRT reasoning translation models were supervised on synthesized long thought and translation data, and reinforcement-learning routes such as R1-T1 and MT-R1 used machine translation metrics like COMET and CometKiwi as reward. Built on Qwen2.5-7B, DeepTrans-7B improves over its backbone by 16.3%, 13.8% and 9.0% on GEA5, GEA100 and GRF on MetaphorTrans, and beats sub-30B baselines on most metrics.
Tencent (China)
Jiaan Wang（First author）
DeepTrans Deep Reasoning Translation Reward Modeling Reinforcement Learning GRPO
arXiv Oct 14, 2026
Robot rewards stop chasing lighting and camera angle: task-relevant dynamics lift MetaWorld success to 94.3%
DiTaR builds video rewards on task-relevant latent dynamics, reaching 94.3% final success on MetaWorld-v3 versus 65.2% for the strongest video-reward baseline ReWiND, and raising real-robot offline RL success from 43.3% to 83.3%.
The University of Texas at Austin Texas A&M University Georgia Institute of Technology
Dayou Li（First author）
DiTaR Reward Modeling Rectified Flow Robotic Manipulation Reward Design
arXiv Sep 30, 2026
LexReward scores Chinese legal responses along Style, Element and Chain, and its reward models lift GRPO's Element average by 6.04 points over SFT
The work introduces LexReward, a taxonomy built with legal experts that decomposes legal response quality into Style (lexical and syntactic), Element (subjects, facts, statutes, decisions) and Chain (order, completeness, correctness, non-redundancy), designs rubrics scored by rules or an LLM judge, uses the resulting rewards to build preference data and train LexRM, a family of Chinese legal reward models; experiments show the rubrics distinguish responses of different quality, DPO improves generation across all three dimensions, LexRM-guided selection beats random selection in test-time scaling, and LexRM used for GRPO yields gains in every dimension and outperforms rule-based outcome rewards.
The work introduces LexReward, a taxonomy built with legal experts that decomposes legal response quality into Style (lexical and syntactic), Element (subjects, facts, statutes, decisions) and Chain (order, completeness, correctness, non-redundancy), designs rubrics scored by rules or an LLM judge, uses the resulting rewards to build preference data and train LexRM, a family of Chinese legal reward models; experiments show the rubrics distinguish responses of different quality, DPO improves generation across all three dimensions, LexRM-guided selection beats random selection in test-time scaling, and LexRM used for GRPO yields gains in every dimension and outperforms rule-based outcome rewards.
Peking University Tsinghua University Northeastern University
Yida Cai（First author）
LexReward LexRM Reward Modeling Rubric-Based Evaluation Direct Preference Optimization
arXiv Sep 30, 2026
TRM first writes a case-adaptive rubric before scoring, beating open-source reward models and nearing proprietary ones on image generation and editing benchmarks
The work introduces the "Think Before You Score" paradigm and the Thinking Reward Model (TRM), which first generates a case-adaptive rubric for each task condition and candidate output, then inspects the candidate criterion by criterion and aggregates the evidence into a fine-grained pointwise reward; it also proposes PD-GRPO to use pairwise preference supervision for better discrimination while mitigating score polarization, and reports state-of-the-art results among open-source reward models on image generation and editing reward-modeling benchmarks, highly competitive with proprietary alternatives, with TRM-guided reinforcement learning consistently improving diverse visual generation models.
The work introduces the "Think Before You Score" paradigm and the Thinking Reward Model (TRM), which first generates a case-adaptive rubric for each task condition and candidate output, then inspects the candidate criterion by criterion and aggregates the evidence into a fine-grained pointwise reward; it also proposes PD-GRPO to use pairwise preference supervision for better discrimination while mitigating score polarization, and reports state-of-the-art results among open-source reward models on image generation and editing reward-modeling benchmarks, highly competitive with proprietary alternatives, with TRM-guided reinforcement learning consistently improving diverse visual generation models.
Xuehai Bai（First author）
Thinking Reward Model Think Before You Score Pairwise Dual-Group Relative Policy Optimization Reward Modeling Image Generation
arXiv Sep 29, 2026
DRM replaces the scalar reward head with a diffusion head: matching or beating same-data, same-backbone baselines on five benchmarks while its reward distribution turns multimodal as human disagreement grows
The work introduces DRM (Diffusion Reward Model), which recasts reward modeling as conditional density estimation: on top of a frozen LLM encoder, a lightweight Diffusion Transformer denoises Gaussian noise into a reward vector with no parametric family assumed, a single architecture handles both multi-attribute regression and pairwise preference data, it matches or surpasses the same-data, same-backbone scalar head ArmoRM and quantile head QRM across five benchmarks, and it exhibits multimodal structure tied to human disagreement on repeated-annotation data, with uncertainty-aware rejection, LCB aggregation, and downstream RLHF experiments demonstrating the practical value of distributional information.
The work introduces DRM (Diffusion Reward Model), which recasts reward modeling as conditional density estimation: on top of a frozen LLM encoder, a lightweight Diffusion Transformer denoises Gaussian noise into a reward vector with no parametric family assumed, a single architecture handles both multi-attribute regression and pairwise preference data, it matches or surpasses the same-data, same-backbone scalar head ArmoRM and quantile head QRM across five benchmarks, and it exhibits multimodal structure tied to human disagreement on repeated-annotation data, with uncertainty-aware rejection, LCB aggregation, and downstream RLHF experiments demonstrating the practical value of distributional information.
Tsinghua University University of Illinois Urbana-Champaign
Xiang Wang（First author）
Diffusion Reward Model (DRM)Reward Modeling Diffusion Transformer Reinforcement Learning from Human Feedback Test-Time Scaling

You’re all caught up

Different LLMs can now collaborate federatedly: FedVP pools locally trained verifiers instead of merging parameters

Literary translation now rewards a large model's scores: reinforcement learning on source sentences alone lifts a 7B model past QwQ-32B-preview on metaphor-heavy text

Robot rewards stop chasing lighting and camera angle: task-relevant dynamics lift MetaWorld success to 94.3%

LexReward scores Chinese legal responses along Style, Element and Chain, and its reward models lift GRPO's Element average by 6.04 points over SFT

TRM first writes a case-adaptive rubric before scoring, beating open-source reward models and nearing proprietary ones on image generation and editing benchmarks

DRM replaces the scalar reward head with a diffusion head: matching or beating same-data, same-backbone baselines on five benchmarks while its reward distribution turns multimodal as human disagreement grows