Research ideation stalls at step one: forward-citation supervision lifts an 8B model to 0.452 graded recall on held-out NLP papers
Synopsis
In automated research ideation, the step of finding a limitation that later work can actually pick up has lacked a direct supervision signal. ActLim-Bench links each source paper's limitation to the later citing work that adopted it as motivation and tried to address it, yielding 18,279 evidence-linked triples. ActLim-8B, post-trained in three stages on that signal, reaches 0.452 graded recall on a temporally held-out 2024–2025 NLP test set against 0.262 for the strongest zero-shot baseline, and transfers from NLP-only training to computer vision at 0.314 versus 0.252.
Interpretation
The first step of the ideation pipeline is trained on its own: given only a source paper's abstract, introduction and conclusion, the model must write three specific limitations that follow-up work could act on. Earlier limitation data came from author-written sections, reviewer critiques or controlled perturbations, which record what authors disclose or reviewers criticize and give limited evidence about which shortcomings later became productive starting points. In a control sharing 1,240 papers, 2,795 examples and about 2.1M target tokens with matched target-length distributions, forward-citation supervision reaches 0.353 graded recall, 0.117 above the strongest alternative; controlled perturbations reach 0.236 and author-stated targets 0.193.
The supervision comes from forward citations: later papers that cite the source state which limitation they adopted and tried to address, turning a subjective judgment into an auditable training target. Each record keeps evidence spans from both the source paper and the citing paper, so an individual target can be checked rather than merely scored. ActLim-Bench holds 18,279 evidence-linked triples over 6,116 papers from 2017 to 2025; two annotators auditing 400 accepted triples found 0.87 precision when all criteria must hold, with 0.88 on Train and 0.86 on Test.
The same verification signal serves in three stages as demonstration targets, a data-selection rule and an online reward over a complete response, and the deployed model needs neither citing papers nor verifier calls. Under a budget matched at 9,000 generated items and 9,000 verifier calls, item-level filtering reaches 0.410 graded recall and response-level reward 0.425, and combining them gives 0.452. On the NLP test set ActLim-8B reaches 0.452 graded recall and 0.241 specific-level recall, against 0.262 and 0.098 for the strongest zero-shot system; in blind human comparisons its limitations are preferred in 64.9% of non-tied judgments.
Perspective
The demonstrated scope is NLP and computer vision conference and journal papers, with the model receiving only the source paper's abstract, introduction and conclusion at inference while citing papers, evidence spans and verifier scores stay hidden; the computer vision set is used only for transfer evaluation and not for training or model selection.
The benchmark records only limitations that follow-up work actually pursued, so it is a proxy for actionability rather than a measure of scientific value; a limitation may go unpursued because of timing, resources or shifting priorities, and a pursued direction is not necessarily a paper's most important shortcoming. Computer vision papers carry fewer observed targets on average than NLP papers (2.05 versus 2.50), so absolute recall values are interpreted within domain.