BaseBench puts every checkpoint of every biological AI model on one leaderboard, with training left running
Synopsis
Biological foundation models have lacked a common, broad benchmark: relevant tasks span nucleic acids, proteins, genomes, cells, whole organisms and clinical work, and lab validation is too slow to run on every checkpoint. BaseBench, being built by Basecamp Research, separates evaluations, checkpoints and adapters so every checkpoint of every model runs the same evaluations, with data version, checkpoint identity, adapter configuration and scoring steps recorded. It already runs on internal and public biological models, with public submissions planned in the near term.
Interpretation
Evaluation of biological AI models moves from fragmented, largely manual work to a single platform that runs the same evaluations on every checkpoint of every model. Biology has lacked the common and broad benchmark sets that more established areas of AI have; relevant tasks each carry their own datasets and targets, and lab validation is far too slow to run on every checkpoint. BaseBench runs every checkpoint of every model, including publicly available biological models, through the same evaluations and testing, so all models are assessed in the same way.
The platform decouples evaluations, checkpoints and adapters so models and evaluations can be added independently. An adapter lets a family of models provide the outputs an evaluation requires, such as embeddings, likelihoods or a generation, and a new checkpoint can reuse its model family's existing adapters. A sensor watches the versioned data store for new checkpoints or changes to evaluation datasets, a compute worker runs the appropriate adapter, a grader scores the output, and the result is recorded in a table that feeds the leaderboard, without pausing training.
Each leaderboard result records the conditions that produced it, so results can be traced and evaluations reproduced. The record covers the version of the data, the identity of the checkpoint, the configuration of the adapter and the steps from input data to final score, and each checkpoint also records its model family and accepted sequence types. When a model cannot provide the output an evaluation requires, BaseBench records that limitation and the reason.
Perspective
BaseBench currently runs on Basecamp Research's own internal models and on publicly available biological models, covering application areas from DNA to the clinic, and public submissions are planned in the near term.
The stated target is hundreds of evaluations across many areas of biology with public submissions planned, but the number of evaluations in each current area is illustrative in the figure; the platform also notes that consistent procedures still require context such as the dataset, its split and the scoring method to interpret a result.