# Agent evaluation

CyberArena turns cybersecurity evaluation from static targets into multi-round combat: two agents in isolated sandboxes attack each other's services while patching their own, with objectives rotating every round. Across 40 challenges and 466 games, GPT-5.6 Sol and Claude Opus 5 led 12 frontier agents, and on 30 challenges all three compared models solved solo, round-three capture rates fell to 52%, 37% and 35%.

CyberArena turns cybersecurity evaluation from static targets into multi-round combat: two agents in isolated sandboxes attack each other's services while patching their own, with objectives rotating every round. Across 40 challenges and 466 games, GPT-5.6 Sol and Claude Opus 5 led 12 frontier agents, and on 30 challenges all three compared models solved solo, round-three capture rates fell to 52%, 37% and 35%.

FinCosmos opens all three uncertainties of financial agent evaluation at once: a simulated investor reveals a hidden strategy only when asked, the agent must source its own Web data and backtest under a data cutoff, and programs verify the delivery instead of matching a reference answer. Across 150 sessions with four coding agents and a tool-free model, none is accepted, and all 35 sessions the simulated user signed off fail at least one program veto.

FinCosmos opens all three uncertainties of financial agent evaluation at once: a simulated investor reveals a hidden strategy only when asked, the agent must source its own Web data and backtest under a data cutoff, and programs verify the delivery instead of matching a reference answer. Across 150 sessions with four coding agents and a tool-free model, none is accepted, and all 35 sessions the simulated user signed off fail at least one program veto.

Agent evaluation has relied on fixed benchmarks that report task success or failure but cannot say whether reviewing or steering an agent mid-trajectory is worth it at deployment. Researchers bring micro-randomized trials into agent trajectories, assigning review or direct execution with equal probability at each eligible decision, then estimate causal effects with augmented inverse probability weighting and structural nested mean models. On τ-bench Retail, ToolSandbox and ImpossibleBench, review effects vary substantially across tool-call types and agent states, and the same data yield decision-group risk rankings.

Agent evaluation has relied on fixed benchmarks that report task success or failure but cannot say whether reviewing or steering an agent mid-trajectory is worth it at deployment. Researchers bring micro-randomized trials into agent trajectories, assigning review or direct execution with equal probability at each eligible decision, then estimate causal effects with augmented inverse probability weighting and structural nested mean models. On τ-bench Retail, ToolSandbox and ImpossibleBench, review effects vary substantially across tool-call types and agent states, and the same data yield decision-group risk rankings.

AstroAgentBench turns space mission planning into an executable, externally checked agent evaluation: 116 cases across seven task families, with the main evaluation using five held-out cases per family, 35 in total, where an agent must submit a schedule, observation plan or constellation design that an independent verifier checks for schema, timing, geometry, resources and mission value. Among five agent systems, Codex CLI + GPT-5.4 and Kimi CLI + Kimi K2.6 produced verifier-valid plans on all 35 cases, and the strongest systems approach or exceed solver-reference scores on several families while still trailing clearly on geometry-heavy, product-level and design-heavy tasks such as Regional Coverage and Stereo Imaging.

AstroAgentBench turns space mission planning into an executable, externally checked agent evaluation: 116 cases across seven task families, with the main evaluation using five held-out cases per family, 35 in total, where an agent must submit a schedule, observation plan or constellation design that an independent verifier checks for schema, timing, geometry, resources and mission value. Among five agent systems, Codex CLI + GPT-5.4 and Kimi CLI + Kimi K2.6 produced verifier-valid plans on all 35 cases, and the strongest systems approach or exceed solver-reference scores on several families while still trailing clearly on geometry-heavy, product-level and design-heavy tasks such as Regional Coverage and Stereo Imaging.

AI research agents must train and evaluate a model for every candidate program, so execution cost caps how many candidates they can compare per step. MLE-World, finetuned from Qwen3.5-4B, reads a program's code and data description and predicts its terminal output, exception type and validation score, plus a confidence score. On held-out programs from 17 MLE-bench Lite tasks its predicted terminal output matches the real run on 77.9% of programs versus 41.1% for the strongest prompted baseline, and search guided by it reaches up to 22.1% higher observed mean performance than the agent without a world model while using less real execution.

AI research agents must train and evaluate a model for every candidate program, so execution cost caps how many candidates they can compare per step. MLE-World, finetuned from Qwen3.5-4B, reads a program's code and data description and predicts its terminal output, exception type and validation score, plus a confidence score. On held-out programs from 17 MLE-bench Lite tasks its predicted terminal output matches the real run on 77.9% of programs versus 41.1% for the strongest prompted baseline, and search guided by it reaches up to 22.1% higher observed mean performance than the agent without a world model while using less real execution.

Frontier AI agent time horizons are usually computed from average success: the task length an agent completes with 50% probability, measured in human expert time. This analysis instead fits a reliability horizon from METR's public run-level records, using the probability that all four attempts on the same task succeed. Across 50 model-benchmark combinations the reliability horizon is always shorter than the capability horizon, with a geometric-mean ratio of about 2.4, and the gap did not narrow significantly between 2023 and early 2026.

Frontier AI agent time horizons are usually computed from average success: the task length an agent completes with 50% probability, measured in human expert time. This analysis instead fits a reliability horizon from METR's public run-level records, using the probability that all four attempts on the same task succeed. Across 50 model-benchmark combinations the reliability horizon is always shorter than the capability horizon, with a geometric-mean ratio of about 2.4, and the gap did not narrow significantly between 2023 and early 2026.

ACED-Bench separates natural-language scientific scenarios from hidden probabilistic graphical models so agents must choose observational or interventional experiments within a four-turn budget and commit to causal answers, producing wide spread across models and agent designs on 180 hard tasks while strong agents near ceiling on a 150-question qualification check.

ACED-Bench separates natural-language scientific scenarios from hidden probabilistic graphical models so agents must choose observational or interventional experiments within a four-turn budget and commit to causal answers, producing wide spread across models and agent designs on 180 hard tasks while strong agents near ceiling on a 150-question qualification check.