Long-horizon agent training stops grouping by repeated states alone: adding task progress to the comparison lifts WebShop success by 3.96 and 5.16 points

When training long-horizon language-model agents, comparing two actions from the same group of sampled trajectories now depends on both the pre-action state and how far the task has already progressed. Grouping by repeated states or transitions previously placed decisions with different prior task progress into one group, so their outcome contrast mixed in earlier progress. Across ALFWorld and WebShop with Qwen2.5-1.5B and 7B, mean success rises over matched GraphGPO by 1.04/1.46 and 1.09/0.47 percentage points, including ALFWorld baselines already above 92%.

The procedure turns existing trajectories into weighted action-pair preferences and adds them as an auxiliary objective alongside shared task learning. Matching forms a comparison context from the pre-action state summary and observed rubric progress, tries ordered exact and coarse rules, and keeps the first rule with evidence for both actions. Labels are oriented by terminal outcomes from the opposite fixed fold and weighted by evidence support; on WebShop with Qwen2.5-1.5B, 1,805 of 1,990 accepted preferences (90.70%) use coarse matching.

The construction also holds on Qwen3 backbones, and each design component has control support. This preference construction had not been tested on Qwen3-4B and 8B. On WebShop, Qwen3-4B rises from 78.18% to 82.14% and Qwen3-8B from 79.69% to 84.84%, with all six paired runs positive; three-seed factorial controls give 2.34–2.40 points from opposite-fold evidence and 1.61–1.67 points from evidence weighting.

The method is evaluated on ALFWorld and WebShop with Qwen2.5-1.5B/7B and Qwen3-4B/8B, three training seeds per setting, with the primary metric averaging the last five scheduled evaluations of a shared 250-update budget. It targets training settings that must extract action-level supervision from existing rollout groups, reusing sampled trajectories and policy log probabilities while preserving critic-free training. WebShop rubric channels use structured training information from the goal scorer, including attribute, option, and price matches and a pre-purchase preview score.

Preferences come from observational outcome contrasts within finite rollout groups; coarse matching can retain outcome-relevant differences, and terminal success depends on decisions after the matched action as well. Evidence weighting only moderates the influence of sparse comparisons, and direct validation of local action preferences remains an important direction. The main comparisons use three training seeds and repeated evaluation on a fixed validation set, so small gains retain statistical uncertainty; the success and integrated-success paired intervals span zero in all four Qwen2.5 settings. Under matched interaction budgets on ALFWorld, the three seed differences are one positive, one zero, and one negative.