184 authors labeled what inspired 207 papers, and the best retriever recovers only 48% of those catalyst papers from a 191K corpus
The authors built ScholarCatalyst, a benchmark in which 184 researchers annotated 894 early-stage research questions and the prior "catalyst" papers that did or could have advanced 207 recent computer science projects, then evaluated retrieval over a 191K-paper corpus restricted to work published before each project; the strongest system reached only 0.48 Recall@20, agentic search (0.42) did not beat embedding retrieval, and even Claude Fable 5.1, whose training data may include the source papers, reached only 0.51.