Can AI agents predict wet-lab results? SPU scores 100 public and 16 unpublished experiments, and the best configuration beats chance on only about half its tasks
Synopsis
Science Prediction Universe (SPU) places AI agents in a real wet-lab prediction setting: given only experimental conditions and candidates, an agent has eight hours to search the literature, run calculations and rank candidates, which are then scored against measurements by Spearman correlation. The 100 public tasks come from recent Nature Portfolio papers and the 16 private tasks from unpublished experiments, so retrieval cannot help. The best of thirteen frontier-model configurations beats chance on only about half its scored tasks, and more reasoning effort or more tool families brought no detectable score gain.
Interpretation
SPU turns predicting a not-yet-run wet-lab experiment into an automatically scored task: each task supplies the candidates, materials and conditions of one study's experiments plus how the result was measured, and the agent must order each comparison group from lowest to highest measured value. Earlier benchmarks either ask a model a single closed question without tools, or let agents write code on data already in hand; wet-lab measurements and hours-long autonomous computation rarely appeared together. The 100 public tasks come from recent papers in 11 Nature Portfolio journals, and 16 private tasks come from unpublished experiments run with partner labs on VHH nanobodies, reaction conditions, electrolytes and variants of the antibody Magrolimab, all measured with replicates.
The agent works alone for up to eight hours in a container with web access and scientific software, and a separate container without network access, which the agent never reaches, scores the submitted ranking against the measurements. The score is the Spearman correlation between submitted and measured order: random rankings average 0 and a perfect ranking scores 1; each task also has a chance threshold, the score only 5% of 4,000 random rankings exceed. Thirteen configurations each ran one official run on every task, giving 1,155 scored runs and 227,103 tool calls on the public tasks; scoring is automatic and no model acts as a judge.
A stable gap separates the leading six configurations from the other seven, and it comes mainly from judgment rather than compute: only 9% of the leading six's runs score below zero, against 22% for the other seven. Where the consensus ranking is right, the leading six keep about 86 of every 100 pairs in measured order and the other seven about 78; where the consensus is wrong, both groups correct only about 30. On the 28 tasks scored by all thirteen configurations, 27 public and one private, the leading six still hold the first six places, in a different order.
The private tasks rule out looking up answers, and twelve configurations still rank significantly better than chance; yet on the same electrolyte series all thirteen rank ionic conductivity almost perfectly (0.89 to 0.95), while on the electrochemical window the twelve scored configurations range from −0.55 to 0.07 and none beats chance. GPT-6 Astra at max reasoning effort runs 4.8 times longer and costs 5.3 times more than at medium effort, with no detectable score gain; its first complete ranking appears at a median 26% of the run and scores 0.469 on average, against 0.424 for the final submission, a difference that is not significant. Of 208 official runs on the 16 private tasks, 159 were scored; a GPU workload raises the score by 0.035 relative to other runs of the same configuration on the same tasks, with a 95% interval from −0.019 to 0.090 that includes zero.
Perspective
The evaluation covers 100 public tasks, 39 in chemistry, 29 in materials science, 20 in biology and 12 in medicine, plus 16 private tasks from unpublished experiments, with agents working in a container with 8 CPU cores, 16 GiB of memory and one NVIDIA H20 GPU for up to eight hours. The complete public task packages, including inputs, environments, scoring code and reference answers, are released on Hugging Face, with reference answers kept in a separate grading container during evaluation. Private tasks stay private for controlled evaluation.
Public tasks come from published papers, so training-data contamination cannot be ruled out entirely, which is why SPU keeps private tasks; on them twelve configurations still beat chance. Claude Opus 5.5 and Claude Fable 5.1 were scored on fewer tasks because of frequent safety refusals, so their averages rest mostly on materials-science tasks. Differences between disciplines are not statistically detectable, since shuffling discipline labels often produces differences this large.