Wenbo Pan

The work defines an agent's taste as its ability to choose the better direction before the outcome is visible, builds Taste-Bench, a 502-question benchmark mined automatically from decision forks in existing agent trajectories, finds that the best model answers only 59.7% correctly while forks whose deciding evidence appears later are much harder and a larger reasoning budget does not help, and shows that distilling the reasoning of a teacher that has seen the outcome into a student improves judgment on unseen tasks by 17.9 percentage points and raises end-to-end success on held-out SWE-bench Pro tasks from 14.6% to 33.7%.

The work defines an agent's taste as its ability to choose the better direction before the outcome is visible, builds Taste-Bench, a 502-question benchmark mined automatically from decision forks in existing agent trajectories, finds that the best model answers only 59.7% correctly while forks whose deciding evidence appears later are much harder and a larger reasoning budget does not help, and shows that distilling the reasoning of a teacher that has seen the outcome into a student improves judgment on unseen tasks by 17.9 percentage points and raises end-to-end success on held-out SWE-bench Pro tasks from 14.6% to 33.7%.