# WebShop

Online Rubric Self-Distillation (ORSD) has the current policy generate natural-language process rubrics from differences among a group of its own trajectories, lets a frozen verifier pick the bundle whose rubric satisfaction best correlates with task success, and uses that bundle as privileged context: the log-likelihood gap for the same tokens with and without the rubric becomes a token-level correction to the trajectory-level GRPO advantage. On ALFWorld and WebShop, Qwen3-1.7B and Qwen3-4B reach the best overall results among the compared methods under the same environment-interaction budget.

Online Rubric Self-Distillation (ORSD) has the current policy generate natural-language process rubrics from differences among a group of its own trajectories, lets a frozen verifier pick the bundle whose rubric satisfaction best correlates with task success, and uses that bundle as privileged context: the log-likelihood gap for the same tokens with and without the rubric becomes a token-level correction to the trajectory-level GRPO advantage. On ALFWorld and WebShop, Qwen3-1.7B and Qwen3-4B reach the best overall results among the compared methods under the same environment-interaction budget.

Long-horizon language-model agent training now builds action preferences inside existing rollout groups, matching on pre-action state plus task progress, and raises mean WebShop success by 3.96 and 5.16 points on Qwen3-4B and 8B.

Long-horizon language-model agent training now builds action preferences inside existing rollout groups, matching on pre-action state plus task progress, and raises mean WebShop success by 3.96 and 5.16 points on Qwen3-4B and 8B.

RELACE is a critic-free framework that uses a frozen behavior policy to teacher-force score each executed action under both its original context and an outcome-augmented context, normalizes and clips the ratio of these length-normalized scores within each trajectory to reweight discounted returns, and then builds local advantages by comparing weighted returns among actions from matched states, combining them with trajectory-level GRPO supervision; on ALFWorld and WebShop with Qwen2.5-1.5B-Instruct and Qwen2.5-7B-Instruct it reports the highest mean aggregate metrics among the compared methods, including GRPO, GiGPO, HCAPO, and GraphGPO.

RELACE is a critic-free framework that uses a frozen behavior policy to teacher-force score each executed action under both its original context and an outcome-augmented context, normalizes and clips the ratio of these length-normalized scores within each trajectory to reweight discounted returns, and then builds local advantages by comparing weighted returns among actions from matched states, combining them with trajectory-level GRPO supervision; on ALFWorld and WebShop with Qwen2.5-1.5B-Instruct and Qwen2.5-7B-Instruct it reports the highest mean aggregate metrics among the compared methods, including GRPO, GiGPO, HCAPO, and GraphGPO.

UniSkill lets a shared policy act as both the actor that executes environment actions and the proposer that emits skillbank edits (Add, Update, No Edit), and trains the proposer with contrastive action feedback that measures how replacing the retrieved skill changes the current actor's action log-likelihood gap between previously collected successful and failed trajectories, avoiding extra rollouts per proposal, with skill-edit support regularization preserving exploration; it reaches 98.4% success on ALFWorld and 84.7% success on WebShop while sustaining stable joint training over 250 steps.

UniSkill lets a shared policy act as both the actor that executes environment actions and the proposer that emits skillbank edits (Add, Update, No Edit), and trains the proposer with contrastive action feedback that measures how replacing the retrieved skill changes the current actor's action log-likelihood gap between previously collected successful and failed trajectories, avoiding extra rollouts per proposal, with skill-edit support regularization preserving exploration; it reaches 98.4% success on ALFWorld and 84.7% success on WebShop while sustaining stable joint training over 250 steps.

Treating the individual instances inside a gated self-evolving LLM agent (DSH-plugin-based, with plugins satisfying four architectural properties: permutation symmetry, reversibility, acyclicity, and typed contracts) as identical hard spheres, the work builds a rigorous layer (any-time hitting-time certificate, resolution law, separation theorem), a kinetic layer (a master equation over the plugin x version x task grid with collision, reaction, external field, and gate operators plus its moment hierarchy), and a measured layer (a faithful minimal instance in a one-model, one-task-family WebShop environment), finding that population fluctuation scaling is density-gated: invisible at sparse edit density and emerging at the predicted rate under tripled density.

Treating the individual instances inside a gated self-evolving LLM agent (DSH-plugin-based, with plugins satisfying four architectural properties: permutation symmetry, reversibility, acyclicity, and typed contracts) as identical hard spheres, the work builds a rigorous layer (any-time hitting-time certificate, resolution law, separation theorem), a kinetic layer (a master equation over the plugin x version x task grid with collision, reaction, external field, and gate operators plus its moment hierarchy), and a measured layer (a faithful minimal instance in a one-model, one-task-family WebShop environment), finding that population fluctuation scaling is density-gated: invisible at sparse edit density and emerging at the predicted rate under tripled density.