Calibrated uncertainty often fails to beat random sampling: active learning on out-of-distribution materials data
Synopsis
Active learning for materials and molecular property prediction treats high-uncertainty samples as the information a model is missing. Across formation energy, bandgap and solubility tasks, uncertainty came from loss-landscape sampling and from ensembles retrained with different seeds, then was calibrated with linear calibration factors and a neural network. Calibrated uncertainty generally did not reduce the data a model needed to improve on out-of-distribution data, showing no advantage over random sampling or uncalibrated estimates, and error bars overlapped.
Interpretation
Picking high-uncertainty samples for active learning does not necessarily beat random sampling on out-of-distribution data. Prior practice often assumed random forests are intrinsically calibrated, or that calibration alone makes uncertainty usable for exploratory sampling. For formation energy with fluorine-containing compounds omitted from training and used as the query pool, random sampling matched or beat uncertainty-maximization methods until about 20% of the out-of-distribution training data was added, and loss-landscape uncertainty only caught up near 50% of the pool.
Uncertainty can come from perturbing one trained model's weights rather than only from repeated retraining. The loss-landscape method interpolates weights along the largest and smallest Hessian eigenvectors, keeps models below a loss threshold of 0.1, then samples a Gaussian approximation of their parameters 50 times to obtain the standard deviation over 500 models' predictions. Across formation energy, bandgap and solubility tasks, this loss-landscape ensemble was compared with a random ensemble of five models trained with different seeds, and both showed low calibration error on in-distribution test data.
A calibration model fitted on in-distribution residuals transfers unevenly to out-of-distribution data. The calibration-factor method assumes a linear relation between uncertainty and residual, while the neural method approximates histogram calibration with a differentiable soft-max. For bandgap, the calibration-factor method collapsed because in-distribution test residuals were non-Gaussian; neural calibration consistently beat calibration factors on in-distribution data, while calibration factors generalized slightly better to out-of-distribution data.
The same pattern reappears for random forest and XGBoost, indicating it is at least partly intrinsic to the data. Tree models used intrinsic model uncertainty and committee disagreement instead of loss-landscape ensembles. For formation energy, random sampling again performed as well as or better than uncertainty maximization for both random forest and XGBoost, consistent with the ALIGNN trend.
Perspective
The findings rest on formation energy and bandgap tasks from the JARVIS 3D DFT dataset, the ESOL solubility task from MoleculeNet, and ALIGNN, random forest, XGBoost and multilayer perceptron models, with in-distribution and out-of-distribution splits built by omitting fluorine or iron and by clustering.
Error bars for the different methods overlap in the active learning curves, which the authors read as showing no single best out-of-distribution acquisition method; the calibration-factor method collapsed for bandgap because in-distribution residuals were non-Gaussian, and the authors also report that calibration had no significant effect on in-distribution or out-of-distribution uncertainty distributions.