CWE-bench v2 opens 100 real-repository vulnerability fixes: the top model still misses 19% on every attempt

The public benchmark offers 100 tasks, each giving an agent a checkout of a real open-source repository and a short instruction to find and fix one security weakness without breaking existing functionality. The setup is the same as before, and v2 expands the public set to 100 tasks covering 91 different CWEs, the numbered classification of security defect types. Scoring is fully programmatic, so results are deterministic and anyone can reproduce them without extra cost; every task was also reviewed by hand and checked for whether the instruction reads like a realistic instruction, whether every graded behaviour can be inferred from it, and whether the verifier checks correct behaviour rather than the structure of one fix.

The task design aims at coverage: doing well requires understanding the multiple weakness surfaces an attacker could exploit, and recognizing a familiar pattern is not enough. A defender has to anticipate every way an attacker could get in and get it right every single time, so fluency in one vulnerability class does not stand in for capability. The 100 tasks span 91 CWEs across a wide range of vulnerability classes; each task also checks that the starter code fails and the reference fix passes, and that verifiers reject memorized solutions such as the known upstream patch.

Of the 11 evaluated models, Gemini 4 Argon leads with 67.8% pass@1 and 81.0% pass@4, just ahead of Claude Opus 5.5 at 67.3% pass@1 and 78.0% pass@4. 95 tasks are solved by at least one model, an indication that the tasks are solvable; however, even the leader fails 19% of the set on every attempt, so the long tail is far from solved. On cost, DeepSeek V4.1 Flash, Claude Opus 5.5 and Gemini 4 Argon form the Pareto frontier: DeepSeek reaches 57.0% pass@1 at $0.04 per rollout, Opus adds ten points at $1.83, and Gemini edges ahead at $3.92.

The public set is 100 tasks covering 91 CWEs with fully programmatic scoring; a separate held-out set tracks tasks no model has seen and requires contacting cyber@collinear.ai to evaluate on it. Tasks are packaged as Harbor tasks and can be run straight from the Hugging Face repository, after accepting the dataset's access conditions on its page.

The leading model fails 19% of the set on every attempt, so the long tail remains unsolved; the public and held-out sets are maintained separately, and the held-out set tracks tasks no model has seen.