Don't write code yet: across 341 repository-level tasks, multi-turn questioning lifts fine-grained requirement alignment from 0.212 to 0.494

Repository-level software development can now be evaluated on the step of clarifying before coding, not only on writing code once requirements are given. Earlier repository-level code generation benchmarks supplied either complete specifications or underspecified requests, and none included multi-turn requirement clarification. The benchmark holds 341 tasks drawn from game, miniapp, and web development, each paired with a human-validated detailed requirement document and corresponding code evaluation artifacts.

Clarification is split into two separately measurable aspects: whether questions elicit new requirements, and whether the final requirement document matches the true requirements. Earlier clarification evaluations focused mainly on function-level tasks, and their simulated users often followed a fixed interaction pattern. Evaluation uses an LLM-based user simulator whose three personas come from a large-scale dataset of multi-turn conversations between real developers and coding agents; two annotators classified 180 interaction trajectories by persona with an average accuracy of 95.8%.

Multi-turn clarification does make requirement documents more complete, especially by recovering fine-grained requirements missing from the initial intent. Compared with generating a requirement document directly from the initial intent, documents produced after multi-turn clarification score higher. Across six models, average BERTScore F1 rose from 0.664 to 0.712 and average requirement alignment rose from 0.212 to 0.494, a 133.1% relative improvement.

The requirement information gathered through clarification carries through to code implementation. Writing code straight from the initial intent and writing it after first producing a requirement document are two different starting points. On the three source benchmarks, scores rise from initial intent to directly generated document to clarified document: GameCraft-Bench 0.012, 0.273, 0.418; MiniAppBench 0.451, 0.543, 0.943; WebGen-Bench 0.271, 0.384, 0.659.

The work examines multi-turn clarification from vague intent to requirement document in repository-level software development, with tasks covering game, miniapp, and web development, and evaluates six LLMs plus the coding-agent harnesses of two of them. The intended readers are teams working on code generation evaluation and requirements engineering. The source proposes that requirement analysis and implementation can be decoupled: use a base model for requirement analysis and document generation, then pass the document to a coding-agent harness for repository-level implementation.

The user is simulated by an LLM, which may not fully capture how real developers differ in expressing needs, disclosing information, and revising requirements; the source mitigates this with three personas and human validation. The source also notes that clarification effectiveness and requirement document quality reflect two related but distinct capabilities, that model performance varies substantially across user personas, and that coding-agent harnesses improve clarification effectiveness without further improving requirement document quality.