Causal agents get graded on experiments, not answers: ACED-Bench hides the graph and spreads 180 hard tasks across models

The evaluation target shifts from whether an answer is right to whether the agent can run an investigation: it must decide which observational or interventional samples to request and then draw a causal conclusion from them. Earlier causal-reasoning benchmarks usually supply the relevant evidence directly as text, a graph, or a fixed dataset, and scientific-agent benchmarks emphasize tool use and multi-step exploration, so few combine active experiment choice with causal decisions in a formally consistent world. The main ACED-Core leaderboard holds 180 four-turn tasks, 60 set-valued causal ancestry/descendancy questions and 120 constrained causal-decision questions, alongside a separate 150-question yes/no qualification check, ACED-Qual, on the same hidden-PGM interface.

Each world is generated by a split of labor between a language layer and a formal layer: an LLM writes variable names, scenarios, and task text, while a programmatic backend builds the graph, samples conditional probability tables, defines intervention semantics, validates the world, and computes the oracle answer. Gold answers are therefore computed programmatically from the known full probabilistic graphical model rather than manually labeled, and generated worlds are accepted only after checks on effect-size margins, threshold gaps, and tie tolerances. The evaluation covers seven contemporary models (Claude Opus-4.6, Claude Sonnet-4.6, Gemini-3-Pro, DeepSeek-V4-Pro, Qwen3.6-27B, GPT-OSS-120B, and Llama-4-Maverick) plus two earlier-generation reference models, across three active agent designs.

Final-answer accuracy hides bottlenecks inside the investigation, and an evidence ledger localizes the failure to a specific stage. Across 3,579 evaluated modular active-code trajectories, failures often begin before the final response: some runs never produce a strong-evidence turn for the target causal contrast, some compute supporting evidence but do not retain it, and some keep querying after sufficient evidence has appeared. The ledger judges each turn on whether it computed the correct estimand, whether the resulting evidence supports the gold answer, and whether that evidence is adequate, marking a turn as strong evidence only when all three hold, which separates experiment selection, evidence retention, ignored evidence, and extra turns after evidence.

The current benchmark uses fully observed categorical probabilistic graphical models with known intervention semantics, spans eight topical domains, and consists of 180 ACED-Core tasks under a four-turn budget plus a 150-question ACED-Qual qualification check. It is intended for evaluating AI systems proposed as scientific assistants and for comparing agents' experiment selection and causal decisions in controlled synthetic worlds; a larger release could support finer analysis by domain, graph structure, decision archetype, and failure mode.

The benchmark does not yet cover continuous variables, latent confounding, temporal dynamics, measurement error, selection effects, model misspecification, or open-ended experiment design. The scenarios are synthetic, so success in these hidden-PGM worlds should not be read as evidence of deployment readiness in real scientific or policy settings; the evidence ledger remains a diagnostic analysis, and the benchmark score is final-answer accuracy under the fixed interaction budget.