Code review agents lifted bug catch rate from 47.7% to 71.9% in five months, but weaken on proprietary repositories

Code review agents can already find bugs that humans confirmed and fixed in real repositories, with the best configuration covering 79.6% of pre-merge bugs and 64.2% of post-merge bugs within its top-5 comments. Earlier code review benchmarks treated human comments as ground truth and expected AI reviews to match them lexically or semantically, even though human comments can be nitpicks accepted through social dynamics or can miss valid points an AI review would raise. CRBugBench contains 1,000 review tasks from 98 open-source GitHub repositories plus 800 tasks from two proprietary repositories at one large company, across five programming languages; tasks are built backward from human-fixed bugs, and 37 of 40 randomly sampled tasks (92.5%) were judged valid.

Review capability rose steadily across model generations, from 47.7% (opus-4.6) to 71.9% (opus-5) averaged over both bug tracks between February and July 2026. The same benchmark and the same copilot-cli review harness measure the generational gap directly, rather than comparing numbers across papers. On pre-merge bugs, claude-opus-5 reaches 79.6% top-5 recall versus 59.1% for claude-opus-4.6; on post-merge bugs the figures are 64.2% and 36.4%.

Agents weaken sharply on bugs that need project context, cross-file reasoning, or runtime evidence, and proprietary repositories contain far more of those bugs. In open-source tasks 46.1% of bugs are directly visible in the diff, versus 22.8% in proprietary tasks; adjusting for bug visibility alone explains 20% of the performance gap. A multivariable logistic regression gives an adjusted odds ratio of 2.621 for diff-visible bugs, 0.454 for non-functional usability bugs, and 1.906 for more severe bugs.

Scaling test-time compute pays off most through prompt changes rather than multi-agent setups: exploration-nudging prompts gain up to 5.8% at about 10% higher cost. The source reports the multi-agent gain inconsistently: the introduction states 4.7% improvement at 5.55x cost, while the RQ4 summary states 2.8% on average at 3.68x cost; raising reasoning effort from low to max lifts gpt-5.6-luna from about 36% to 57% at 23.22x the cost. For gpt-5.6-sol, the comprehensive prompt raises pre-merge top-5 recall from 65.6% to 71.3% and post-merge from 49.0% to 54.0%, at roughly the same cost.

The benchmark measures only review agents' recall of known bugs, not overall review usefulness; tasks come from mature, active open-source projects and two large proprietary industry repositories, covering Python, Rust, C#, JavaScript/TypeScript, and Java. The authors recommend that teams apply the same data-construction method to build custom benchmarks for their own repositories and periodically re-evaluate their review configuration as models evolve.

The authors state that the regressions estimate adjusted associations rather than causal effects, and that unmeasured factors and LLM annotation errors may confound or attenuate estimates; benchmark construction can still yield incorrect ground truths or unsolvable tasks; agent behavior is stochastic and sensitive to changes; and results favor mature, active projects, so they may differ for smaller or younger projects, uncommon languages, or configuration-heavy code.