A decision model that acts badly can still teach: JEV scores whole trajectories and lifts a 1.5B policy from 72.8% to 82.9%
Synopsis
Decision models are usually judged by how well they act, yet the same models are pulled toward an easier subgoal when acting on ALFWorld. Used instead as a teacher that sees the whole trajectory after an episode and is asked whether each action helped, JEV raises a Qwen2.5-1.5B policy trained with GRPO from 72.8% to 82.9% success on ALFWorld and from 56.8% to 71.4% on WebShop, while JEV as an actor completes only 42.9% of ALFWorld tasks.
Interpretation
A decision model's value lies not only in choosing the right action but in scoring actions already taken and using those scores as a training signal. Prior work studied these models mainly as actors and measured their usefulness by acting success; as actors, all six tested models are pulled away from a subgoal that must come first by an easier one. On ALFWorld, JEV as an actor completes 42.9% of tasks overall, 70.8% of tasks with two independent subgoals, and only 14.7% of tasks whose subgoals must be completed in order.
Using JEV as a teacher, adding its per-action scores to GRPO's outcome reward improves a small policy's success rate. Trained on outcome rewards alone, every action in a trajectory receives the same credit, so the steps that actually advanced the task are not distinguished. The Qwen2.5-1.5B policy rises from 72.8% to 82.9% on ALFWorld and from 56.8% to 71.4% on WebShop; asking JEV instead whether the task will succeed reaches only 69.3% on ALFWorld.
Whether teacher scores help depends on the question and the visible context: ask about each action's contribution and show the whole trajectory. With only the trajectory prefix visible, the two questions train similarly; once the full trajectory is visible, the questions separate. Across 462 ALFWorld trajectories, asking whether an action helped scores milestone actions 0.36 higher than no-op actions, while asking whether the task will succeed gives a gap of only 0.05.
Perspective
The findings rest on two text environments, ALFWorld and WebShop, with episodes of at most 30 actions, and the teacher JEV stays frozen during training and is not queried at test time. The authors list software engineering, tool use and assistant tasks with longer trajectories as an open question.
JEV and GPT Decisions are closed models with undisclosed adaptation procedures, so which training choices make a decision model a good judge cannot be determined. Judging happens after an episode ends; when a judge is used online to filter or rerank actions, only the prefix is available, a setting not yet covered.