Skip to main content

# Long-Horizon Agent Evaluation

2 related articles

Subscribe to updates

arXiv Oct 8, 2026
AgentTime benchmark: GPT-6 Astra ends 63% of runs on time in Codex, but some on-time runs sleep after appearing to finish
The authors introduce AgentTime, a benchmark of 222 tasks from 18 sources run in native CLI harnesses to test duration-following, runtime forecasting, and retrospective time estimation; GPT-6 Astra in Codex ends 63% of runs within 5% of the requested duration versus 4% for Fable 5.1, 14 of 158 classifiable Astra runs explicitly slept after appearing to finish, forecasts mostly overestimate natural runtimes, and stripping time cues grows retrospective misses about two- to fivefold.
The authors introduce AgentTime, a benchmark of 222 tasks from 18 sources run in native CLI harnesses to test duration-following, runtime forecasting, and retrospective time estimation; GPT-6 Astra in Codex ends 63% of runs within 5% of the requested duration versus 4% for Fable 5.1, 14 of 158 classifiable Astra runs explicitly slept after appearing to finish, forecasts mostly overestimate natural runtimes, and stripping time cues grows retrospective misses about two- to fivefold.
ELLIS Institute Tübingen
Michael Ofengenden（First author）
AgentTime Duration Following Long-Horizon Agent Evaluation LLM Agents Claude Code
arXiv Sep 24, 2026
The Tasteful Agent: Measuring and Training Long-Horizon Judgment from Trajectory Hindsight
The work defines an agent's taste as its ability to choose the better direction before the outcome is visible, builds Taste-Bench, a 502-question benchmark mined automatically from decision forks in existing agent trajectories, finds that the best model answers only 59.7% correctly while forks whose deciding evidence appears later are much harder and a larger reasoning budget does not help, and shows that distilling the reasoning of a teacher that has seen the outcome into a student improves judgment on unseen tasks by 17.9 percentage points and raises end-to-end success on held-out SWE-bench Pro tasks from 14.6% to 33.7%.
The work defines an agent's taste as its ability to choose the better direction before the outcome is visible, builds Taste-Bench, a 502-question benchmark mined automatically from decision forks in existing agent trajectories, finds that the best model answers only 59.7% correctly while forks whose deciding evidence appears later are much harder and a larger reasoning budget does not help, and shows that distilling the reasoning of a teacher that has seen the outcome into a student improves judgment on unseen tasks by 17.9 percentage points and raises end-to-end success on held-out SWE-bench Pro tasks from 14.6% to 33.7%.
Microsoft (United States)City University of Hong Kong
Wenbo Pan（First author）
Taste-Bench Long-Horizon Agent Evaluation Knowledge Distillation LLM Agents

You’re all caught up