Multi-turn agents stop learning only from the final score: recurring low-reward decisions get the extra penalty, and ALFWorld hits 97.9% at 4B

The training signal for a multi-turn agent no longer comes only from the terminal reward of a whole trajectory; it can land on a specific intermediate decision. Group-relative advantages previously made every token in one trajectory share the same reward-based comparison, so they said little about which intermediate decisions in a long interaction to adjust; actor-critic methods such as PPO can estimate state-dependent values but need an extra critic and still do not label locally correct or incorrect actions. On AppWorld, PEARL improves over its ARPO backbone on both Normal and Challenge tasks at both 4B and 8B scales, by 3.67 and 1.16 points at 4B and by 4.56 and 1.40 points at 8B.

It strengthens negative advantages only for decisions that recur on lower-reward branches and have no matching support on successful branches, leaving positive advantages and the base policy optimizer unchanged. The mechanism starts from branching rollouts: when two continuations from the same history end with different rewards, it locates their first actual token divergence and represents that decision by concatenating the later-layer logits for the chosen token, a compact signature rather than the much larger final hidden state. Failure and success features go into separate banks clustered with HDBSCAN, the success bank asymmetrically vetoes ambiguous failure clusters, and a sliding window keeps only the most recent 25 updates; on ALFWorld at 4B the mean success rate is 97.9%, 2.7 points above the strongest baseline, GiGPO.

The report is for researchers and engineers training multi-turn language agents, and covers AppWorld Normal and Challenge tasks and ALFWorld Seen and Unseen tasks with Qwen3-4B-Instruct-2507 and Qwen3-8B, trained for 400 optimizer updates on AppWorld and 300 on ALFWorld, with three training seeds in the main experiments. The method builds on ARPO and depends on its branching rollouts to supply matched same-history comparisons; ablations run on AppWorld with Qwen3-4B-Instruct-2507 as a single controlled run evaluated at step 400.

Retained patterns are candidates for targeted updates, not labels of intrinsically incorrect actions, and their value comes from combining a local outcome contrast with recurrence and a check against successful behavior. At 8B, PEARL's 97.1% ALFWorld mean exceeds ARPO by 1.7 points but trails GiGPO by 0.8 points. Under unlimited history, one retained failure cluster at update 400 contains 1,200 of 1,368 failure entries, while at updates 100 and 300 success-bank filtering leaves no retained failure cluster, so bank size alone does not guarantee a match with current decisions.