Deep Learning Foundation Models for Low-Data Regimes from Classical Molecular Descriptors
This study introduces a new avenue for foundation model pretraining: supervised pretraining on low-noise, calculable molecular descriptors to obtain rich, highly transferable molecular representations, demonstrated with CheMeleon, an O(10M) parameter foundation model; across 58 benchmark data sets spanning properties relevant to small-molecule drug discovery and sourced from the industry-led Polaris benchmarking initiative, CheMeleon enables directed message-passing neural networks to finally exceed classical methods in the low-data regime, outperforms classical baselines such as Random Forest on molecular fingerprints and descriptors as well as existing foundation models under rigorous statistical comparisons, and the model and pretraining framework are open-sourced.