Coding agents stop asking everything: filtering for questions that change behavior lifts F1 by 13.88 points across four agents
Synopsis
Coding agents often fill in underspecified requirements with their own assumptions, producing code that looks correct but behaves in ways the user never intended, and fixing that later grows costly. CONTRA splits clarification into broad question generation followed by semantic filtering and execution-based qualification, keeping only questions whose two plausible answers yield stable behavioral differences, then uses interaction history to ask or stop. On ClarifyCodeBench it reaches the highest F1 with all four coding agents, 13.88 percentage points above the best baseline macro-average.
Interpretation
Clarification in code generation shifts from asking whenever a question arises to asking only questions that change program behavior, so agents no longer silently resolve unspecified requirements through their own assumptions. Existing methods struggle to identify key clarification questions while avoiding unnecessary ones, and unnecessary questions interrupt developers and slow development. On ClarifyCodeBench, CONTRA achieves the highest F1 with all four coding agents, exceeding the best baseline macro-average F1 by 13.88 percentage points.
Candidate questions are first generated broadly, then filtered for relevance to required behavior and for whether the requirement already resolves them. Each surviving question is judged at the behavioral level rather than by its wording or by model self-assessment. For each remaining question, programs are generated conditioned on two plausible answers and checked for stable behavioral differences on shared inputs; only questions producing such differences qualify.
Interaction history then decides whether to ask among qualified questions or to stop asking. With the same LLM and evaluation protocol, CONTRA achieves higher clarification recall and F1 than the coding harnesses Claude Code and OpenHands. The method is implemented as a Claude Code plugin that integrates selective clarification into everyday development.
Perspective
This report draws on summary-level information, not the complete paper. The demonstrated scope is evaluation on ClarifyCodeBench: CONTRA achieves the highest F1 with all four coding agents, 13.88 percentage points above the best baseline macro-average, and higher clarification recall and F1 than Claude Code and OpenHands under the same LLM and evaluation protocol. The method is implemented as a Claude Code plugin for integrating selective clarification into everyday development.
The summary gives no per-agent breakdown and does not state behavior outside the benchmark. The 13.88-point gap is in macro-average F1, and its statistical stability and variation across LLMs remain unclear.