# Reward Design

11 related articles

Subscribe to updates

arXiv Nov 10, 2026
Reward design for clinical multi-turn agents: outcome-only training shortens the work-up, dense question credit keeps high-risk cases
Clinical multi-turn agents must decide what to ask, when to stop, and what to do from incomplete information. Across three controlled simulated settings, medication refill triage, acute diagnostic work-up and dosing titration, the study compares reward designs: outcome-only GRPO often raises average value by asking fewer questions while producing premature commitment and weaker high-risk performance, whereas extra credit for questions that reduce uncertainty about the hidden state raises held-out value and moderate-case support across three training seeds in diagnostics without the high-risk seed collapse seen under outcome-only training.
Clinical multi-turn agents must decide what to ask, when to stop, and what to do from incomplete information. Across three controlled simulated settings, medication refill triage, acute diagnostic work-up and dosing titration, the study compares reward designs: outcome-only GRPO often raises average value by asking fewer questions while producing premature commitment and weaker high-risk performance, whereas extra credit for questions that reduce uncertainty about the hidden state raises held-out value and moderate-case support across three training seeds in diagnostics without the high-risk seed collapse seen under outcome-only training.
Brendan P. Murphy（First author）
Safe Reinforcement Learning GRPO Credit Assignment Multi-turn agents Reward Design
arXiv Oct 14, 2026
Agents iterate their own data recipes: math and code selection strategies gain up to 13.34% under fixed training
Agents use experimental feedback to revise data-engineering strategies, and across four tasks—math and code data selection, math reward design, and math curriculum learning—the selected strategies gain up to 13.34% absolute over generic baselines on AIME25 Pass@32 under fixed training conditions.
Alibaba Group (China)University of North Carolina at Chapel Hill Peking University
Ruomeng Ding（First author）
Agentic Data Engineering Data Selection Reward Design Curriculum Learning
arXiv Oct 14, 2026
Robot rewards stop chasing lighting and camera angle: task-relevant dynamics lift MetaWorld success to 94.3%
DiTaR builds video rewards on task-relevant latent dynamics, reaching 94.3% final success on MetaWorld-v3 versus 65.2% for the strongest video-reward baseline ReWiND, and raising real-robot offline RL success from 43.3% to 83.3%.
The University of Texas at Austin Texas A&M University Georgia Institute of Technology
Dayou Li（First author）
DiTaR Reward Modeling Rectified Flow Robotic Manipulation Reward Design
arXiv Oct 12, 2026
One RNA design policy now serves many backbones: pairing MCC climbs from 0.61 to 0.73
Structure-conditioned RNA inverse design previously either trained a separate policy per target backbone or relied on offline preference optimization, leaving online post-training of a reusable backbone-conditioned model largely unexplored. MORAD maps six design measurements to bounded desirability scores, combines them with a weighted geometric mean, and uses group-relative normalization to drive a forward-process diffusion update. On 153 evaluation targets it raises GDT-TS from 0.3300 to 0.3827 and pairing MCC from 0.6110 to 0.7295, while cutting normalized ensemble defect from 0.3906 to 0.2830.
Structure-conditioned RNA inverse design previously either trained a separate policy per target backbone or relied on offline preference optimization, leaving online post-training of a reusable backbone-conditioned model largely unexplored. MORAD maps six design measurements to bounded desirability scores, combines them with a weighted geometric mean, and uses group-relative normalization to drive a forward-process diffusion update. On 153 evaluation targets it raises GDT-TS from 0.3300 to 0.3827 and pairing MCC from 0.6110 to 0.7295, while cutting normalized ensemble defect from 0.3906 to 0.2830.
Fudan University
Jixin Tang（First author）
MORAD RNA Inverse Folding Reinforcement Learning DiffusionNFT Reward Design
arXiv Oct 8, 2026
How Retrieval Credit Is Assigned Decides Search-Agent Training Gains: Local Credit Lifts Qwen3-4B Average F1 from 51.25 to 54.34
The study builds corpus-grounded QA data with intermediate evidence annotations and compares three retrieval signals (evidence coverage Cov, coverage with dependencies Cov-Dep, answer match AM) under scalar reward bonuses versus action-aligned local credit, training Qwen3-4B on seven QA benchmarks and finding local credit outperforms scalar, with Cov-local reaching 54.34 average F1 versus 51.25 for outcome-only training.
The study builds corpus-grounded QA data with intermediate evidence annotations and compares three retrieval signals (evidence coverage Cov, coverage with dependencies Cov-Dep, answer match AM) under scalar reward bonuses versus action-aligned local credit, training Qwen3-4B on seven QA benchmarks and finding local credit outperforms scalar, with Cov-local reaching 54.34 average F1 versus 51.25 for outcome-only training.
University of Edinburgh
Wenyu Huang（First author）
Credit Assignment Reward Design GRPO Reinforcement Learning with Verifiable Rewards Agentic Search
arXiv Oct 6, 2026
BIRD Unifies Iterative Reward Design Evaluation: Simple Design Choices Are Most Robust, and ERA-U/ERA-S Rank First and Second Under Matched Budgets
The authors introduce BIRD, which expresses 14 published iterative reward design (IRD) methods as configurations of one loop, compares 6 published methods and ablates 30 design choices across MuJoCo, Meta-World, Assistax and HumanoidBench under matched feedback conditions and policy-training budgets, finding that most choices (including the authors' own proposals) have small or inconclusive effects while a few simple choices (such as greedily retaining checkpoints and summarizing feedback) help consistently; a small hill-climb from RDA yields ERA-U and ERA-S, which take first and second place in the pooled Bradley–Terry ranking, though a small human study shows high benchmark scores do not imply natural behavior.
The authors introduce BIRD, which expresses 14 published iterative reward design (IRD) methods as configurations of one loop, compares 6 published methods and ablates 30 design choices across MuJoCo, Meta-World, Assistax and HumanoidBench under matched feedback conditions and policy-training budgets, finding that most choices (including the authors' own proposals) have small or inconclusive effects while a few simple choices (such as greedily retaining checkpoints and summarizing feedback) help consistently; a small hill-climb from RDA yields ERA-U and ERA-S, which take first and second place in the pooled Bradley–Terry ranking, though a small human study shows high benchmark scores do not imply natural behavior.
University of Edinburgh Anthropic (United States)
Logan Mondal Bhamidipaty（First author）
Large Language Model Reward Hacking Vision-Language Model Judging Reward Design Benchmark Construction and Evaluation
arXiv Oct 3, 2026
X-Tree mines reusable skills from trajectories into a tree, lifting success by up to 4.5%, 5.8% and 4.1% on WebArena, ScienceWorld and WebShop at matched data and budget
The work introduces X-Tree: raw actions are first canonicalized into typed tokens such as type⟨date⟩ or click⟨button⟩, then adjacent pairs are recursively merged under an X-Score that weighs recurrence, span length and the fraction of occurrences in successful episodes, yielding a deterministic reusable experience tree mined from a trajectory pool with zero LLM calls; X-Tree is then integrated into three training settings — offline RL with each node as a training instance under a step-matching reward plus a depth-scaled completion bonus, online RLVR with an adaptive skill bonus that supplies signal while the verifier is silent and anneals as it becomes informative, and on-policy self-distillation where a rendered X-Tree replaces the LLM-written skill bank as the self-teacher's privileged c
The work introduces X-Tree: raw actions are first canonicalized into typed tokens such as type⟨date⟩ or click⟨button⟩, then adjacent pairs are recursively merged under an X-Score that weighs recurrence, span length and the fraction of occurrences in successful episodes, yielding a deterministic reusable experience tree mined from a trajectory pool with zero LLM calls; X-Tree is then integrated into three training settings — offline RL with each node as a training instance under a step-matching reward plus a depth-scaled completion bonus, online RLVR with an adaptive skill bonus that supplies signal while the verifier is silent and anneals as it becomes informative, and on-policy self-distillation where a rendered X-Tree replaces the LLM-written skill bank as the self-teacher's privileged c
Duke University University of Waterloo National University of Singapore
Sitao Cheng（First author）
X-Tree GRPO Reward Design On-Policy Self-Distillation
arXiv Oct 2, 2026
InterEvolve evolves reward programs at test time for a fixed humanoid controller, lifting simulated success from 34.6% to 86.5% and running autonomously on a Unitree G1
InterEvolve introduces a test-time evolution framework in which an object-aware forward-backward (FB) behavioral foundation model executes reward programs, an LLM agent revises program structure in context while CMA-ES tunes its constants, and every candidate is verified across parallel simulation scenarios, letting a fixed humanoid controller solve untrained loco-manipulation tasks without retraining and raising macro-average success across eight task families from 34.6% for the best calibrated fixed program to 86.5%, with evolved skills deployed on a physical Unitree G1.
InterEvolve introduces a test-time evolution framework in which an object-aware forward-backward (FB) behavioral foundation model executes reward programs, an LLM agent revises program structure in context while CMA-ES tunes its constants, and every candidate is verified across parallel simulation scenarios, letting a fixed humanoid controller solve untrained loco-manipulation tasks without retraining and raising macro-average success across eight task families from 34.6% for the best calibrated fixed program to 86.5%, with evolved skills deployed on a physical Unitree G1.
Zhuo Lin（First author）
InterEvolve Test-Time Evolution of Reward Programs Humanoid Loco-Manipulation Reward Design Foundation Models
arXiv Sep 28, 2026
RULER uses per-instruction six-item rubrics as RL rewards, lifting Qwen3-8B SVG generation from 0.432 to 0.693 and matching the much larger DeepSeek-V3
The work first shows, on 900 human-rated SVG samples generated by Claude-Opus-4.6, Qwen3-32B, and Qwen3-8B, that prompting a vision-language judge with a multi-axis rubric correlates with human judgment far better than scalar metrics such as CLIP and aesthetic scores (Spearman 0.7929, Goodman-Kruskal Gamma 0.7574), then introduces RULER: it derives a six-item instance-aware rubric from the text instruction alone, has a judge VLM score each rendered rollout item-by-item, and optimizes the weighted reward with GRPO, raising the rubric score on MMSVG-Illustration and MMSVG-Icon from the Qwen3-8B backbone's 0.432 and 0.395 to 0.693 and 0.683, surpassing dedicated SVG specialists and matching the substantially larger DeepSeek-V3 without paired SVG ground truth or human preference labels.
The work first shows, on 900 human-rated SVG samples generated by Claude-Opus-4.6, Qwen3-32B, and Qwen3-8B, that prompting a vision-language judge with a multi-axis rubric correlates with human judgment far better than scalar metrics such as CLIP and aesthetic scores (Spearman 0.7929, Goodman-Kruskal Gamma 0.7574), then introduces RULER: it derives a six-item instance-aware rubric from the text instruction alone, has a judge VLM score each rendered rollout item-by-item, and optimizes the weighted reward with GRPO, raising the rubric score on MMSVG-Illustration and MMSVG-Icon from the Qwen3-8B backbone's 0.432 and 0.395 to 0.693 and 0.683, surpassing dedicated SVG specialists and matching the substantially larger DeepSeek-V3 without paired SVG ground truth or human preference labels.
The Hong Kong University of Science and Technology (Guangzhou)University of Oxford
Hangyu Ran（First author）
RULER SVG Code Generation Rubric-Based Evaluation GRPO Reward Design
ACM Conference on Recommender Systems (RecSys) Sep 25, 2026
Google Deploys a Dual-Loop LLM Agent System on YouTube That Autonomously Found RMSprop, Gated Path, and a Multi-Objective Reward, Lifting Multiple A/B Metrics
The work proposes and deploys a self-evolving recommendation system in which Gemini-family LLM agents act as machine learning engineers: an Offline Agent (Fast Loop, waking every 5 minutes and running daily) generates and offline-scores candidate configurations, while an Online Agent (Slow Loop, running daily) ranks candidates and drives A/B experiments; across several YouTube recommendation surfaces the agents autonomously discovered optimizer switches (Adagrad to RMSprop, FTRL), a Gated Path (GLU) architecture, activation refinement, multi-objective reward synthesis, and reward hyperparameter tuning, with multiple online metrics improving at the 95% confidence level and experiment throughput rising from Θ(1)-Θ(10) per week to Θ(100) per week.
The work proposes and deploys a self-evolving recommendation system in which Gemini-family LLM agents act as machine learning engineers: an Offline Agent (Fast Loop, waking every 5 minutes and running daily) generates and offline-scores candidate configurations, while an Online Agent (Slow Loop, running daily) ranks candidates and drives A/B experiments; across several YouTube recommendation surfaces the agents autonomously discovered optimizer switches (Adagrad to RMSprop, FTRL), a Gated Path (GLU) architecture, activation refinement, multi-objective reward synthesis, and reward hyperparameter tuning, with multiple online metrics improving at the 95% confidence level and experiment throughput rising from Θ(1)-Θ(10) per week to Θ(100) per week.
Google
Self-Evolving Recommendation System LLM Agents Recommender systems Reward Design