# Credit Assignment

12 related articles

Subscribe to updates

arXiv Nov 4, 2026
Agent RL stops scoring whole trajectories: ORSD writes its own rubrics to assign credit token by token
Online Rubric Self-Distillation (ORSD) has the current policy generate natural-language process rubrics from differences among a group of its own trajectories, lets a frozen verifier pick the bundle whose rubric satisfaction best correlates with task success, and uses that bundle as privileged context: the log-likelihood gap for the same tokens with and without the rubric becomes a token-level correction to the trajectory-level GRPO advantage. On ALFWorld and WebShop, Qwen3-1.7B and Qwen3-4B reach the best overall results among the compared methods under the same environment-interaction budget.
Online Rubric Self-Distillation (ORSD) has the current policy generate natural-language process rubrics from differences among a group of its own trajectories, lets a frozen verifier pick the bundle whose rubric satisfaction best correlates with task success, and uses that bundle as privileged context: the log-likelihood gap for the same tokens with and without the rubric becomes a token-level correction to the trajectory-level GRPO advantage. On ALFWorld and WebShop, Qwen3-1.7B and Qwen3-4B reach the best overall results among the compared methods under the same environment-interaction budget.
Zhejiang University Chinese Academy of Sciences
Shumao Wu（First author）
Online Rubric Self-Distillation Credit Assignment Self-Distillation GRPO WebShop
arXiv Oct 26, 2026
Long-horizon agents should bank credit per turn: MPO splits turn-level advantage from token refinement, lifting search pass@1 by up to 4.33 points
Long-horizon LLM agents interact with their environment through complete turns but generate each action token by token, and existing reinforcement learning either gives a whole trajectory one shared advantage or treats every token as an action. MPO splits credit assignment into two scales: turn-level generalized advantage estimation carries credit across interactions, a mean-preserving local correction refines it within each turn, and the policy update uses complete-turn importance ratios. Across search, tool-use and automation tasks, MPO reports the highest score on all nine metrics at both Qwen3-4B-Thinking and Qwen3-30B-A3B-Instruct scales, including a 4.33-point BrowseComp-Plus pass@1 gain over the strongest non-MPO baseline at 4B.
Long-horizon LLM agents interact with their environment through complete turns but generate each action token by token, and existing reinforcement learning either gives a whole trajectory one shared advantage or treats every token as an action. MPO splits credit assignment into two scales: turn-level generalized advantage estimation carries credit across interactions, a mean-preserving local correction refines it within each turn, and the policy update uses complete-turn importance ratios. Across search, tool-use and automation tasks, MPO reports the highest score on all nine metrics at both Qwen3-4B-Thinking and Qwen3-30B-A3B-Instruct scales, including a 4.33-point BrowseComp-Plus pass@1 gain over the strongest non-MPO baseline at 4B.
Peking University University of Hong Kong Tsinghua University Renmin University of China
Tao Ren（First author）
Multiscale Policy Optimization Multiscale Advantage Estimation Agentic Reinforcement Learning Credit Assignment
arXiv Nov 10, 2026
Reward design for clinical multi-turn agents: outcome-only training shortens the work-up, dense question credit keeps high-risk cases
Clinical multi-turn agents must decide what to ask, when to stop, and what to do from incomplete information. Across three controlled simulated settings, medication refill triage, acute diagnostic work-up and dosing titration, the study compares reward designs: outcome-only GRPO often raises average value by asking fewer questions while producing premature commitment and weaker high-risk performance, whereas extra credit for questions that reduce uncertainty about the hidden state raises held-out value and moderate-case support across three training seeds in diagnostics without the high-risk seed collapse seen under outcome-only training.
Clinical multi-turn agents must decide what to ask, when to stop, and what to do from incomplete information. Across three controlled simulated settings, medication refill triage, acute diagnostic work-up and dosing titration, the study compares reward designs: outcome-only GRPO often raises average value by asking fewer questions while producing premature commitment and weaker high-risk performance, whereas extra credit for questions that reduce uncertainty about the hidden state raises held-out value and moderate-case support across three training seeds in diagnostics without the high-risk seed collapse seen under outcome-only training.
Brendan P. Murphy（First author）
Safe Reinforcement Learning GRPO Credit Assignment Multi-turn agents Reward Design
arXiv Oct 16, 2026
WebCoder trains frontend agents on a living project: 72.8 requirement satisfaction beats Claude-Opus-4.8's 65.1
WebCoder-27B, trained during an internship at Zhipu AI, keeps reproduction, generation, modification and repair on one evolving project state and scores reinforcement learning by how well the executable application meets task requirements. On the new Webdev-Bench its requirement satisfaction rises from 64.8 for the Qwen3.8-27B base model to 72.8, above Claude-Opus-4.8 at 65.1; on Vision2Web its average visual and functional score rises from 41.1 to 62.4, on par with Claude-Opus-4.8 at 62.5.
WebCoder-27B, trained during an internship at Zhipu AI, keeps reproduction, generation, modification and repair on one evolving project state and scores reinforcement learning by how well the executable application meets task requirements. On the new Webdev-Bench its requirement satisfaction rises from 64.8 for the Qwen3.8-27B base model to 72.8, above Claude-Opus-4.8 at 65.1; on Vision2Web its average visual and functional score rises from 41.1 to 62.4, on par with Claude-Opus-4.8 at 62.5.
Beihang University University of Oslo Tsinghua University
Zijun Dou（First author）
WebCoder Artifact-Grounded Agentic Reinforcement Learning Webdev-World Webdev-Bench GRPO
arXiv Oct 26, 2026
Diffusion-model RL stops scoring whole answers: part-of-speech credit lifts MATH500 by 9.0 points over the base model at the smallest budget
Diffusion language models commit tokens over many denoising steps, so a whole-response reward says little about which tokens helped later refinement. DTCA estimates each token's contribution from the confidence gain at still-masked positions after that token is committed, averages these signals by part-of-speech category into stable weights, and uses them to pick denoising timesteps and assign token-level learning signals, with entropy-based calibration. On LLaDA-8B-Instruct it beats the base model at generation budgets of 128, 256 and 512, by 9.0, 6.8 and 4.8 points on MATH500 and 5.3, 2.1 and 2.5 points on GSM8K.
Diffusion language models commit tokens over many denoising steps, so a whole-response reward says little about which tokens helped later refinement. DTCA estimates each token's contribution from the confidence gain at still-masked positions after that token is committed, averages these signals by part-of-speech category into stable weights, and uses them to pick denoising timesteps and assign token-level learning signals, with entropy-based calibration. On LLaDA-8B-Instruct it beats the base model at generation budgets of 128, 256 and 512, by 9.0, 6.8 and 4.8 points on MATH500 and 5.3, 2.1 and 2.5 points on GSM8K.
Hongze Chen（First author）
Denoising Token Credit Assignment Credit Assignment Diffusion Language Model Reinforcement Learning Mathematical Reasoning
arXiv Oct 12, 2026
Long-horizon agent training stops grouping by repeated states alone: adding task progress to the comparison lifts WebShop success by 3.96 and 5.16 points
Long-horizon language-model agent training now builds action preferences inside existing rollout groups, matching on pre-action state plus task progress, and raises mean WebShop success by 3.96 and 5.16 points on Qwen3-4B and 8B.
Meituan (China)Tsinghua University Imperial College London
Chenxu Zhang（First author）
Matched-Context Group Policy Optimization Credit Assignment LLM Agents ALFWorld WebShop
arXiv Oct 26, 2026
Finer process rewards are not automatically better: decaying the weight lifts long-horizon trailer editing by 45.4%, while a fixed weight truncates 44.1% of rollouts
On four trailer-editing task types built from 49 real films and their released trailers, decaying an LLM-judged turn-level process reward to zero within the same GRPO setup gives a 45.4% relative gain over outcome-only GRPO on Fine Cut, while the fixed-weight variant scores below outcome-only GRPO on the three shorter task types and truncates 44.1% of rollouts. Process rewards do encourage visual inspection and reasoning early, but continued optimization drives redundant tool calls without a committed result; token-level on-policy self-distillation proves highly sensitive to the teacher's privileged information.
On four trailer-editing task types built from 49 real films and their released trailers, decaying an LLM-judged turn-level process reward to zero within the same GRPO setup gives a 45.4% relative gain over outcome-only GRPO on Fine Cut, while the fixed-weight variant scores below outcome-only GRPO on the three shorter task types and truncates 44.1% of rollouts. Process rewards do encourage visual inspection and reasoning early, but continued optimization drives redundant tool calls without a committed result; token-level on-policy self-distillation proves highly sensitive to the teacher's privileged information.
University of Pennsylvania
Lei Zhao（First author）
Credit Assignment Video Editing GRPO On-Policy Self-Distillation Reward Hacking
arXiv Oct 14, 2026
Multi-turn agents stop learning only from the final score: recurring low-reward decisions get the extra penalty, and ALFWorld hits 97.9% at 4B
PEARL turns the first divergence between two same-history branches with different outcomes into a reusable decision memory, strengthens only the negative advantages of recurring low-reward decisions, and reaches 97.9% mean success on ALFWorld at 4B, 2.7 points above the strongest baseline.
Fudan University Shanghai Innovation Institute
Jie Yang（First author）
PEARL (Process-Experience Aware Reinforcement Learning)Credit Assignment LLM Agents AppWorld ALFWorld
arXiv Oct 8, 2026
GraphOPD reweights distillation supervision with an environment-state dependency graph, beating the strongest baseline by up to +5.8 pp on ALFWorld and WebShop
GraphOPD reads which steps enabled which later ones from the environment's own record of state changes, builds a directed dependency graph, scores each step by a random-walk stationary distribution, and fuses that structural credit with the teacher-student divergence into a trajectory-relative mask that distills only each rollout's above-average-aptitude steps; across ALFWorld, WebShop, and SearchQA at three model scales against eleven baselines it stays competitive throughout, improving over the strongest baseline by up to +5.8 pp.
GraphOPD reads which steps enabled which later ones from the environment's own record of state changes, builds a directed dependency graph, scores each step by a random-walk stationary distribution, and fuses that structural credit with the teacher-student divergence into a trajectory-relative mask that distills only each rollout's above-average-aptitude steps; across ALFWorld, WebShop, and SearchQA at three model scales against eleven baselines it stays competitive throughout, improving over the strongest baseline by up to +5.8 pp.
Hong Kong University of Science and Technology Xiaohongshu (Little Red Book) (China)The Hong Kong University of Science and Technology (Guangzhou)
Bohan Lin（First author）
GraphOPD On-Policy Distillation Credit Assignment LLM Agents GRPO
arXiv Oct 7, 2026
RELACE scores actions by comparing original-context and outcome-conditioned likelihoods, reaching the highest reported mean success among compared methods on ALFWorld and WebShop
RELACE is a critic-free framework that uses a frozen behavior policy to teacher-force score each executed action under both its original context and an outcome-augmented context, normalizes and clips the ratio of these length-normalized scores within each trajectory to reweight discounted returns, and then builds local advantages by comparing weighted returns among actions from matched states, combining them with trajectory-level GRPO supervision; on ALFWorld and WebShop with Qwen2.5-1.5B-Instruct and Qwen2.5-7B-Instruct it reports the highest mean aggregate metrics among the compared methods, including GRPO, GiGPO, HCAPO, and GraphGPO.
RELACE is a critic-free framework that uses a frozen behavior policy to teacher-force score each executed action under both its original context and an outcome-augmented context, normalizes and clips the ratio of these length-normalized scores within each trajectory to reweight discounted returns, and then builds local advantages by comparing weighted returns among actions from matched states, combining them with trajectory-level GRPO supervision; on ALFWorld and WebShop with Qwen2.5-1.5B-Instruct and Qwen2.5-7B-Instruct it reports the highest mean aggregate metrics among the compared methods, including GRPO, GiGPO, HCAPO, and GraphGPO.
Columbia University
S. Chakrabarti（First author）
Retrospective Likelihood-based Action Credit Assignment Credit Assignment LLM Agents GRPO ALFWorld