Coding agents hunt bugs on their own, but PROBE finds the best configuration recovers only 20.8% of the bugs maintainers fixed
The PROBE benchmark uses 2,457 bugs that maintainers fixed over 18 months in snapshots of 20 GitHub repositories as a reference, and asks whether coding agents can find those bugs autonomously across a whole repository. Across ten frontier models under a partitioned audit harness, the best configuration recovers only 20.8% of the reference bugs with precision no higher than 8.1%, yet an independent panel confirms 84% of findings that do not match the reference set as real bugs, showing that agents find a different distribution of bugs than maintainers fix.