Can AI code survive review? Cognition built 36 open-source repos into a merge test, and top models still fail over 40% of attempts

Coding-agent evaluation shifts from whether tests pass to whether a maintainer would merge the change, with diff size, out-of-scope edits, and unhelpful tests counted in the score. Earlier coding evaluations mainly looked at test outcomes, while model changes often still needed substantial cleanup before a maintainer would accept them. FrontierCode was built with maintainers of 36 open source repositories who spent more than 40 hours on each task, and the tasks are private to avoid contamination.

Grading combines deterministic tests with qualitative review, and every check in an attempt is either blocking or non-blocking. Passing hidden tests alone is not enough, because failing any blocking check scores that attempt 0. The verifier combines hidden tests and command checks, limits on which files and how much code a change touches, and an LLM review against the maintainers' criteria; an attempt that passes all blocking checks scores between 70% and 100%.

Under one shared agent harness, leading models still fail more than 40% of attempts to clear every blocking check. Claude Opus 5.5 (max with fallback) leads at 52% average score with a 58% pass rate, followed by GPT-6.1 Sol (max) and Claude Fable 5.1 (max with fallback) at 49%. The 100-task Main set is run with three attempts per task, every model runs in the Stirrup harness capped at 250 turns, and GPT-6 Luna (medium) serves as judge; 9 of the 100 tasks are passed by no model on any attempt.

The evaluation covers the 100 tasks of the FrontierCode 1.1 Main set with three attempts per task, with every model running in the Stirrup agent harness capped at 250 turns and GPT-6 Luna (medium) as judge. Tasks come from maintainers of 36 open source repositories and stay private to avoid contamination. The benchmark launches as a standalone leaderboard and is not part of the Artificial Analysis Intelligence Index, with plans to add it to the Coding Agents leaderboard soon.

Pass rate is the share of attempts completing all blocking criteria, while scores also reflect desirable non-blockers, so the two mean different things. Cost and output tokens vary widely, from $0.09 to $12.00 per task and from about 27k to about 236k output tokens, with time per task ranging from about 3 minutes to about 46 minutes. 9 of the 100 tasks are passed by no model on any attempt, and no task is passed by every model on every attempt.