LLM pairwise comparison of historical proposals at three Spallation Neutron Source beamlines ranks them in positive correlation with human ranking, matches human reviewers at flagging high-publication-potential proposals, and costs over two orders of magnitude less
Using historical general user proposals from three beamlines at Oak Ridge National Laboratory's Spallation Neutron Source (EQ-SANS, CNCS, POWGEN), the study has large language models judge every pair of proposals within a run cycle, converts the win-lose outcomes into rankings with the Bradley-Terry model, and finds that LLM rankings correlate positively with human rankings (Spearman ρ about 0.2-0.8, rising to ≥0.5 after 10% outlier removal), show no statistically significant difference from human review in identifying proposals with high publication potential, cost over two orders of magnitude less, and support linear-complexity proposal similarity analysis via embedding models.