Frontier models already design high-risk mutations: 21.4% land in the top risk decile without tools, 23.5% with biological design tools

A public benchmark now measures the high-risk mutation designs a large model actually submits, not just whether it recalls dangerous knowledge. Earlier biosecurity evaluations mostly used question-answering to test whether a model holds hazardous information, or confined tools to stages unrelated to the malicious objective, so the risk amplification from tools went unquantified. The evaluation covers 15 frontier models and 30 scenarios distilled from biosecurity guidelines, each paired with published deep mutational scanning measurements as a comparison.

In each scenario the model receives a wild-type sequence and a malicious objective, and may call six biological design tools to propose, evaluate and revise candidate substitutions. Malicious design is rarely a one-shot process; it is iterative and governed by fitness trade-offs, so the evaluation treats the model as a planner that orchestrates tools rather than a static answerer. Three settings are compared: S0 one-shot, S1 one round of tool revision, and S2 up to four iterative rounds, all taken from one shared generation trajectory with the same initial candidate pool.

Submitted substitutions are scored directly against hidden wet-lab measurements, with no biological material synthesized. Earlier validation relied either on unreliable model judges or on expensive wet-lab synthesis that carries biosafety risk. Scoring draws on 187,695 published in vitro and in vivo measurements in ProteinGym and ViroGym, and 83.9% of S2 submissions on average match a measured record.

The benchmark is aimed at biosecurity evaluators and model developers, and it measures the single amino-acid substitutions models submit across 30 scenarios against published deep mutational scanning measurements. The unit is a single substitution because large-scale causal phenotype labels exist almost only for single amino-acid changes, and the tools likewise return single-site evidence. The authors leave more dangerous multi-mutant or live-organism experiments outside the evaluation, and state that the framework and benign components will be open-sourced while adversarial prompts and hidden ground-truth mappings for the most severe pathogen scenarios will be gated.

The authors state that this version does not run two mechanism controls: shuffled tool outputs, and a non-LLM classical optimizer over the same tool outputs. The workflow's risk increase is therefore supported, but whether the rise comes from the numeric content of tool output rather than the extra deliberation a tool round induces remains unsettled. The no-tool iteration arm is too noisy to serve as a reliable ablation and is read only as a bound, not as evidence that iteration alone has no effect. The unit is a single substitution, so the metric does not estimate multi-mutant fitness, epistasis, host-context expression, transmissibility, or the success of a physical construction pipeline, and no claim is made about behavior in a living organism.