An agent's time horizon shrinks by half when every retry must succeed, and the gap has not closed across model generations
Synopsis
Frontier AI agent time horizons are usually computed from average success: the task length an agent completes with 50% probability, measured in human expert time. This analysis instead fits a reliability horizon from METR's public run-level records, using the probability that all four attempts on the same task succeed. Across 50 model-benchmark combinations the reliability horizon is always shorter than the capability horizon, with a geometric-mean ratio of about 2.4, and the gap did not narrow significantly between 2023 and early 2026.
Interpretation
The time horizon used to describe what an agent can do now has a companion that asks whether it succeeds every time. A deployer delegating a recurring task cares about success on each appearance, while the common 50% horizon describes average success. Prior horizon work calibrated capability by average success against task length, and repeated-trial reliability work measured consistency on fixed tasks without a calibrated length; the two lines had not been joined. On METR's public run-level time-horizon records, the reliability horizon is shorter than the capability horizon in all 50 model-version combinations, and every bootstrap interval of the ratio excludes one.
The distance between the reliability horizon and the capability horizon is substantial and has not visibly shrunk with model generations. With four consecutive successes required, the geometric-mean ratio of capability to reliability horizon is 2.39 (95% interval 2.07–2.84) in benchmark version 1.1 and 2.33 (1.98–2.81) in version 1.0. In version 1.1, Claude Opus 4.6 has a capability horizon of about 715 minutes and a four-attempt reliability horizon of about 355 minutes; GPT-5.2 has about 354 and 192 minutes. The reliability horizon doubles every 3.68 months against 4.02 months for the capability horizon, and the interval on the slope difference includes zero.
Repeated attempts on a task are strongly correlated rather than independent draws. Under independence, the four-attempt horizon would sit where average success is about 84.1%, below the 80% horizon; the observed reliability horizon instead lies between the 80% and 50% horizons. An intraclass correlation puts 68% to 87% of outcome variance between tasks rather than within them (median 0.79 in both versions), and the ratio of observed to independence-implied reliability horizon excludes one in 49 of 50 combinations.
Inconsistency concentrates near each model's capability frontier rather than on the longest tasks. Far below the capability horizon agents almost always succeed on every attempt, and far above it they almost always fail; mixed outcomes occur in fewer than 4% of cells at both extremes. For tasks within a factor of four of the capability horizon the mixed-outcome share rises to 31–40%, the quadratic term is negative and significant in both versions (p < 10⁻⁵), and the peak sits at 0.86 times the horizon in version 1.1 and 0.69 times in version 1.0.
Perspective
The conclusions rest on METR's public run-level time-horizon records, whose task suite is dominated by software and research-engineering tasks, and the horizons describe model-scaffold combinations rather than models alone.
Attempts per model-task cell are modest (median four in version 1.1), limiting the precision of individual cell estimates; human completion times carry measurement error and some long tasks use estimated rather than observed times; the trend analysis covers 20 and 30 models over three years, so power to detect a narrowing gap is limited, and the non-significant narrowing should not be read as a constant gap.