Agents must ask real clients what they left unsaid: across 49 projects, the best delivery score is 0.5510

There is now a test set for whether an agent can turn an underspecified real business request into a deliverable solution. Earlier optimization benchmarks typically begin from explicit objectives, while clarification benchmarks often add information gaps to existing tasks. The 49 FDE-Bench cases span combinatorial optimization and machine learning, and the clarification targets are distilled from discussions between frontline practitioners and forward-deployed engineers.

The agent has to question a simulated customer for the missing requirements, then repeatedly optimize the final artifact. Scoring does not stop at answer correctness: customer-accepted forward-deployed engineer solutions serve as quality references, and case-specific evaluators check final artifacts against acceptance criteria. Across 18 agent configurations the highest mean delivery score is only 0.5510 against the FDE reference of 1.0, and the highest delivery success rate is only 6.12%, the fraction of planned runs reaching that reference.

The benchmark targets end-to-end delivery settings where an agent must clarify requirements in dialogue with a customer and iterate to acceptance criteria; its readers are researchers and engineers studying agent delivery capability. It contains 49 cases spanning combinatorial optimization and machine learning, with clarification targets distilled from discussions between frontline practitioners and forward-deployed engineers, and evaluation uses customer-accepted forward-deployed engineer solutions as quality references.

Among the 18 evaluated agent configurations, the highest mean delivery score is only 0.5510 and the highest delivery success rate is only 6.12%, so reliable FDE-level end-to-end delivery remains out of reach for the evaluated agents. These conclusions are limited to the underspecified real-world business tasks and agent configurations covered by this benchmark.