Agent RL stops scoring whole trajectories: ORSD writes its own rubrics to assign credit token by token
Synopsis
Online Rubric Self-Distillation (ORSD) has the current policy generate natural-language process rubrics from differences among a group of its own trajectories, lets a frozen verifier pick the bundle whose rubric satisfaction best correlates with task success, and uses that bundle as privileged context: the log-likelihood gap for the same tokens with and without the rubric becomes a token-level correction to the trajectory-level GRPO advantage. On ALFWorld and WebShop, Qwen3-1.7B and Qwen3-4B reach the best overall results among the compared methods under the same environment-interaction budget.
Interpretation
In agent reinforcement learning, which intermediate decisions to reinforce and which to suppress can now be signaled per token, instead of every token in a trajectory sharing one advantage value. Under trajectory-level GRPO, all policy-generated tokens in a trajectory share the same advantage, so useful actions, redundant steps and recoverable mistakes receive the same learning signal; GiGPO estimates step-level advantages by grouping actions from matching environment states, but tokens within each action still share one signal. On ALFWorld, ORSD's overall success exceeds the strongest baseline, GRPO, by 12.12 and 3.36 percentage points at the two model scales (Qwen3-1.7B and Qwen3-4B); on WebShop, success exceeds the strongest baseline by 2.80 points over GiGPO with Qwen3-1.7B and by 10.90 points over GRPO with Qwen3-4B.
The method uses the same trajectories twice: the policy first writes process rubrics from behavioral differences across the group, then the rubric is added as extra context and the same tokens are rescored, and the likelihood gap becomes that token's credit correction. Rubrics are generated online by the current policy rather than authored by humans or produced offline by a stronger model and held fixed; outcome rewards are not given as input during generation, so the rubrics do not collapse into a restatement of the reward. At each training step the policy samples a group of trajectories for one task and independently generates four candidate rubric bundles of 5 to 8 criteria each; a frozen verifier (Qwen3.5-Plus via the DashScope API) scores every criterion, and the bundle with the highest Spearman correlation with group outcome rewards is selected, with replay adding no environment interaction.
Candidate rubrics vary in quality, so selection uses the rank agreement between rubric satisfaction and task success rather than picking a bundle at random. Removing rubric-reward alignment and using a single generated bundle directly lowers WebShop success, and so does random selection with the same candidate count. On WebShop, full ORSD improves success over the single-bundle variant by 2.15 and 9.45 percentage points on Qwen3-1.7B and Qwen3-4B; with the candidate count fixed, alignment-based selection improves success over random selection by 3.40 and 7.15 points, with score gains of 5.36 and 4.08 points.
Perspective
The demonstrated scope is Qwen3-1.7B and Qwen3-4B on the two long-horizon interactive benchmarks ALFWorld and WebShop, trained on four A100 80GB GPUs with eight tasks and eight rollouts per task per update for 300 steps and evaluated with the official metrics.
Rubric knowledge is not explicitly accumulated across updates, since rubrics are generated only from the current rollout group with no persistent repository of earlier criteria; selecting bundles by outcome correlation favors rubrics that separate stronger trajectories but does not directly optimize their usefulness for credit assignment within a trajectory.