Coding agents now set each other's exam: GPT-5.5 fixes 98.3% of defects, Claude Opus 4.8 writes the hardest tasks

Evaluation tasks for coding agents are no longer written in advance by humans; the two competing agents author and solve each other's tasks at evaluation time. Earlier coding benchmarks were fixed at release, with tasks and reference patches drawn from public sources, so they leak into the training data of the models they evaluate and are eventually saturated by stronger agents. In a 1,500-turn tournament among 11 model-harness participants on 15 repositories in five languages, GPT-5.5 with Codex CLI repaired 98.3% of the defects it reviewed, and Claude Opus 4.8 with mini-swe-agent reached the field's highest Red win rate at 27.3%.

Each turn has a Red agent add a feature to a real repository and hide a subtle defect inside it, while Blue receives only the public pull request and must find and fix the defect without being told one exists. Red works across three context-isolated sessions: implement the feature with feature tests, embed the defect with sealed bug-triggering tests, and then have a fresh session of the same agent repair the defect blind, so tasks that cannot be repaired are rejected. A task is admitted only if it passes the feature tests and the repository's existing tests while failing exactly the sealed bug tests; in a stratified recheck of 44 tasks, 91.7% were repaired blind in all three re-runs.

The arena targets researchers and engineering teams evaluating repository-level coding agents, and runs on 15 relatively small open-source repositories in five languages, each pinned to a specific commit, with all sessions in isolated containers. A participant is a model-harness pair, so the same model under two harnesses counts as two participants. A newcomer authors its own task bank and plays active-sampled incumbents, reaching a coherent placement in five matches without changing the existing order.

Evidence that task difficulty rises with authoring capability is preliminary: three versions of one model family produced fixed-solver failure rates of 8.3%, 0.0%, and 42.9%, with few tasks per rung and almost fully overlapping intervals for the two older versions, so the trend is not strictly monotone per version. Admission rests on a single blind self-repair run and is therefore stochastic; the sealed bug tests are written by Red, and in a manual audit of 50 turns lost only on bug tests, three were false negatives and two ambiguous. The arena is currently limited to 15 lightweight repositories and may saturate as frontier solvers converge.