Long-horizon agent testbed grows to 77 tasks: outcome-only scoring drops most of the progress

Long-Horizon-Terminal-Bench 2.0 expands the task set to 77 tasks across 13 domains and eight task types, and stretches a single evaluation from 90 minutes to three hours with verifier checkpoints recorded every 30 minutes. The earlier Long-Horizon-Terminal-Bench held 46 tasks with a 90-minute evaluation horizon and only a final-state pass check. Version 2.0 keeps the 46 original tasks, corrects task and evaluation-infrastructure defects, extends 31 of them, and generates the added tasks through a systematic reconstruction-task pipeline.

Scoring no longer reduces to pass or fail: dense progress ladders reward partial progress while blocking memorization of public outputs. Under outcome-only evaluation, intermediate steps that were done correctly contribute nothing to the score. On the 30 reconstruction tasks, binary verifier scores average 0.221 while dense ladder scores average 0.653.

The evaluation covers 1,145 scored model-task runs from 15 frontier models, with GPT-6-astra leading at dense mean reward 0.69 and GPT-5.6-sol at 0.67. Alongside dense mean reward, the count of tasks reaching R at least 0.95 separates steady completion from occasional passes. GPT-6-astra reaches that threshold on 37 tasks and GPT-5.6-sol on 32.

This report draws on summary-level material and does not cover the complete paper. The demonstrated scope is LHTB 2.0 keeping 46 original tasks while correcting defects and extending 31 of them, for 77 tasks across 13 domains and eight task types, with a three-hour horizon and a verifier checkpoint every 30 minutes, plus dense mean rewards and threshold task counts over 1,145 runs from 15 frontier models. It suits researchers and engineers who care about evaluating sustained agent execution.

The summary does not describe the parameterized reference implementation inside the reconstruction-task pipeline, nor does it give per-domain or per-task-type breakdowns. The gap between dense ladder scores and binary scores is reported only on the 30 reconstruction tasks; whether other task types show the same gap is not stated.