Coding agents hunt bugs on their own, but PROBE finds the best configuration recovers only 20.8% of the bugs maintainers fixed
Synopsis
The PROBE benchmark uses 2,457 bugs that maintainers fixed over 18 months in snapshots of 20 GitHub repositories as a reference, and asks whether coding agents can find those bugs autonomously across a whole repository. Across ten frontier models under a partitioned audit harness, the best configuration recovers only 20.8% of the reference bugs with precision no higher than 8.1%, yet an independent panel confirms 84% of findings that do not match the reference set as real bugs, showing that agents find a different distribution of bugs than maintainers fix.
Interpretation
Coding agents can report bugs autonomously across an entire repository, but they recover only a small share of the bugs maintainers actually fix. Earlier bug-discovery evaluations either pointed agents at files or patches known to contain a flaw or focused on security vulnerabilities; whole-repository discovery measured against maintainer fixes had not been systematically evaluated. On 2,457 reference bugs across 20 Python, JavaScript and TypeScript repositories, the best configuration of ten models reaches 20.8% recall, and no configuration exceeds 8.1% precision.
Most bugs agents report are real; they are simply not the same bugs maintainers fix. The reference set contains only bugs fixed within the 18-month window, so low precision alone cannot show that findings are false positives. An independent panel sampled 100 findings per model for nine models and confirmed 84% of findings that do not match the reference set as real bugs; extrapolating fix rates caps a perfect finder's precision at 26–47% against all bugs.
Agents and maintainers differ systematically in bug type, and agent runs overlap with each other more than with maintainer fixes. Agents over-report local correctness errors, while maintainers more often fix incomplete input handling and errors in the public API surface. Wrong arithmetic or comparison expressions make up 18.5% of agent findings versus 9.9% of maintainer fixes; missing branches for a supported input variant make up 7.7% of agent findings versus 18.0% of maintainer fixes; GPT-5.6 Luna runs overlap with each other a median of 6.3 times more than with maintainer fixes within the same bug type.
Perspective
The evaluation covers 20 Python, JavaScript and TypeScript repositories snapshotted on 1 January 2025, with a reference set of 2,457 bugs fixed by maintainers over the following 18 months; every configuration uses the same partitioned audit harness, with each pass run in a separate container with network access and web search disabled.
The reference set is incomplete, since bugs still unfixed at the end of the window are missing, so recall is measured against that set; three models without a published knowledge cut-off cannot be ruled out as having training data covering the reference window; other audit harnesses could yield different results.