Scientific foundation models can now be audited automatically: AutoSPECTRA exposes failure conditions that aggregate scores hide across four biological models

The generalization limits of scientific foundation models can now be audited automatically by a coding agent under shared standards, rather than waiting for scattered user reports after release. Earlier generalizability studies used disparate comparisons and standards of validation, so similar performance patterns could yield different conclusions depending on whether an analysis separated model-specific weaknesses from greater task difficulty. Across four biological foundation models, the audits linked performance variation to protein sequence composition and relatedness to pretraining data for ESMFold2, representation of perturbations in training for STATE, effective spatial resolution for CONCH, and residue-level genomic, evolutionary and structural context for AlphaMissense.

The protocol splits an audit into five stages carried out in sequence by one coding agent, not by several agents. The five stages are the Distiller, Investigator, Dataset Scout and Fetcher, and Auditor, and they represent audit steps rather than separate agents. The Distiller turns a question into a plan naming the target model, prediction task, unit of analysis, performance measure and candidate axes; the Investigator measures the input property, sets at least three ordered levels and fixes the analysis plan before seeing target-model predictions, then runs fixed reference methods independent of the target model on the same examples.

A finding counts only when the performance decline survives checks for alternative explanations and reproduces on data not used to select the axis. Without a prescribed axis or analysis plan, agents proposed literature-grounded hypotheses but none tested a candidate axis empirically; when explicitly asked to quantify a relationship and test it on independent data, all 24 runs completed a quantitative evaluation and 22 reproduced a reported effect. In the CONCH audit, reduced effective resolution lowered CONCH top-1 accuracy from 0.803 at native resolution to 0.276 at 4.0 μm pixels, while an ImageNet-pretrained ResNet-50 on the identical transformed images fell from 0.795 to 0.564, relative losses of 65.6% and 29.1%.

The audits cover four biological domains, protein structure prediction, cellular perturbation modeling, computational pathology and variant-effect prediction, with one full audit and several independent audits per model; the literature synthesis recovers recurring analytical practices rather than establishing a universal definition of a generalizability audit.

The four case studies examine the protocol across domains rather than benchmark auditing performance, so they do not support estimates of audit sensitivity, specificity or false-discovery rate; the experiments were not designed to isolate the effects of agent family, model capability or coding harness, and frontier-agent training histories cannot be reconstructed, so prior exposure to a model, dataset or biological concept cannot be excluded.