CyberArena puts AI agents in live attack-and-defense combat, and two frontier models pull ahead of the field

Cybersecurity agent evaluation moves from single-role work against static targets to two agents attacking and defending in the same game. Earlier benchmarks mostly had an agent face a fixed vulnerable target alone, and scored defense by replaying pre-recorded exploits rather than installing a live adversary. Across 40 challenges and 466 valid games, 12 frontier model–harness entrants were ranked by expected score over a common reference of opponents and challenges; GPT-5.6 Sol led at 78.6, Claude Opus 5 followed at 73.8, and third-place GLM-5.3 scored 68.5.

In each game, two entrants host identical vulnerable services in isolated sandboxes, exploiting the opponent while repairing their own service. The format follows attack–defense capture-the-flag: three 20-minute rounds, no vulnerability hints, objectives rotating each round, and defensive service changes persisting across rounds. Health checks confirm that repairs preserve service functionality, and a latent-ability model adjusts every capture for opponent strength and challenge difficulty to estimate attack, defense and availability abilities.

Under adversarial pressure, challenges that were solved solo regain discriminative power. In solo mode, 30 of the 40 challenges were solved by all three models compared: GPT-5.6 Sol, Claude Opus 4.8 and GLM-5.3. In contested games on those 30 challenges, by round three GPT-5.6 Sol sustained a 52% capture rate, while Claude Opus 4.8 and GLM-5.3 dropped to 37% and 35%.

The evaluation covers 12 fixed model–harness combinations and 40 challenges, 14 adapted from attack–defense CTF competitions and 26 based on open-source software; standings are relative to this roster and challenge set. The platform code, curation pipeline and 40 challenge environments are shared, and game trajectories will be released after review and sanitization.

The trajectory audit was performed by language models with independent agent reviews and selective trace checks, which does not constitute independent human verification of every finding. Integrity probes found six accepted captures through an unauthenticated spectator endpoint before it was restricted, and the audit directly linked six accepted captures, 0.45% of 1,336 accepted captures, to replaying signed requests or reading objective journals; those captures remain in the standings. Passing health checks do not always establish service availability, and the audit recorded rare cases where a defense served the checker while denying other clients.