Ask before you code: 341 repository-level tasks show clarification lifts requirement alignment by 133%
Synopsis
For repository-level software development, researchers built ReqClarifyBench, pairing 341 tasks with vague user intents, human-validated requirement documents, and code evaluation artifacts, and ran multi-turn clarification with simulated users carrying three real-world personas. Across six LLMs, asking effectively did not mean writing better documents: the strongest questioner scored lower on requirement alignment, yet adding interactive clarification raised average alignment from 0.212 to 0.494, a 133.1% relative gain, with code-generation scores rising too.
Interpretation
Repository-level code generation evaluation now treats asking first as its own stage, instead of letting a model guess a full specification from a vague request. Earlier repository-level benchmarks either supplied complete, clear requirement documents or gave vague requests with no multi-turn clarification, so a failure could not be attributed to code ability or to misread requirements. ReqClarifyBench holds 341 tasks, 140 from GameCraft-Bench, 100 from MiniAppBench, and 101 from WebGen-Bench, covering game, miniapp, and web development, each paired with a human-validated requirement document and code evaluation artifacts.
Clarification is split into two separately measurable dimensions: whether questions hit the point, and whether the final requirement document is accurate and complete. Prior clarification studies mostly stayed at function-level tasks and used simulated users following a fixed interaction pattern, which cannot reflect differences among real users. The study measures process with Effective Question Rate and AvgClarified, and outcome with BERTScore F1 and atomic-level Requirement Alignment; the simulator is driven by three personas from SWE-chat, and two annotators classified 180 interaction trajectories with 95.8% average accuracy and Cohen's kappa of 0.892.
The gain from interactive clarification lands mainly on fine-grained requirement alignment and carries through to later code implementation. Generating a requirement document directly from the initial intent left models at an average Requirement Alignment of only 0.212, meaning many details were absent from the initial intent. With multi-turn clarification, the six models' average BERTScore F1 rose from 0.664 to 0.712 and average Requirement Alignment from 0.212 to 0.494, a 133.1% relative gain, with per-model gains from 83.8% to 241.3%; generating code with Codex and GPT-5.6-Sol raised GameCraft-Bench from 0.012 to 0.418, MiniAppBench from 0.451 to 0.943, and WebGen-Bench from 0.271 to 0.659.
Coding-oriented agent harnesses ask more effective questions but do not carry that advantage into the final requirement document. Against their base models, Claude Code and Codex achieved higher Effective Question Rate and AvgClarified under all three personas, yet produced lower-quality final documents. Under the Expert Nitpicker persona, using a coding-agent harness lowered Requirement Alignment by 15.2% for Claude and 24.0% for GPT, with the same trend under the Vague Requester and Mind Changer personas.
Perspective
The evaluation targets repository-level software development, with tasks drawn from game, miniapp, and web scenarios, and interactions carried out by an LLM-based user simulator with three personas, capped at 20 rounds per session. It measures how well models turn vague intent into requirement documents and how those documents affect later code generation; the code-generation experiments use Codex with GPT-5.6-Sol and follow each source benchmark's original evaluation protocol.
The simulated user is an LLM and may not fully capture how real developers express, disclose, and revise requirements; the study mitigates this with three SWE-chat personas and blind human evaluation. Conclusions rest on six models and three personas, with clarification capped at 20 interaction rounds, so behavior beyond that range is not covered.