MOF-Verify uses a failure-aware agentic harness to improve MOF hypothesis verification and releases a four-family diagnostic benchmark

The paper introduces a diagnostic benchmark for metal-organic framework (MOF) hypothesis verification with four task families: structural grounding, synthesis-condition verification, evidence-sufficiency verification, and MLIP-based computational verification. Verification reliability in the MOF setting had lacked systematic characterization; the benchmark decomposes verification into separately examinable task families so that failures in different stages can be localized individually. Per the abstract, the benchmark comprises four task families, with T-MOF-1-3 evaluated under closed-book, retrieval-enabled, and oracle-evidence settings and T-MOF-4 evaluating computational verification separately; the abstract reports no sample sizes or specific metric values.

By evaluating T-MOF-1-3 under closed-book, retrieval-enabled, and oracle-evidence settings, the authors localize failures to knowledge access, evidence acquisition, and reasoning. This setting comparison separates 'the model does not know' from 'the model cannot obtain evidence' from 'the model reasons incorrectly,' which a single end-to-end evaluation cannot do. The abstract states the three settings are used to localize failures in knowledge access, evidence acquisition, and reasoning; no per-setting scores are reported.

Guided by these diagnosed failure modes, the authors develop MOF-Verify, a failure-aware agentic harness that addresses structural, literature, evidence-sufficiency, and computational bottlenecks before producing a final verdict. Unlike direct inference or retrieval-only baselines, the framework converts diagnosed failure modes into pre-verdict processing steps rather than relying on a single generation. The abstract states the framework targets structural, literature, evidence-sufficiency, and computational bottlenecks before producing a final verdict, a method-level description.

Across multiple backbone LLMs, MOF-Verify substantially improves hypothesis-verification performance over direct inference and retrieval-based baselines, and the benchmark datasets are released. The improvement is reproduced across multiple backbone models, indicating the gain is not tied to a single model, and the public dataset release supports follow-up reproduction and extension. The abstract states 'Across multiple backbone LLMs, MOF-Verify substantially improves hypothesis-verification performance over direct inference and retrieval-based baselines' and that 'Benchmark datasets are released'; no effect sizes or statistical tests are given.

The work targets hypothesis-verification pipelines for MOFs and applies to research and engineering settings that need to distinguish knowledge-access, evidence-acquisition, and reasoning failure sources, such as the verification module of an AI-driven materials Co-Scientist. Its four benchmark task families and three evaluation settings provide a reusable evaluation framework for separately improving structural grounding, synthesis conditions, evidence sufficiency, and MLIP computational verification; the released datasets let other teams compare different agent designs on the same task families. For readers who want to use LLMs as materials-reasoning components, the framework offers an organization that localizes bottlenecks before issuing a verdict.

Because only the abstract is available here, the sample size per task family, the specific evaluation metrics, the magnitude of improvement, and statistical significance cannot be determined, nor can the MLIP model and computational conditions used in the computational-verification component be confirmed. The abstract does not describe how oracle evidence is sourced or constructed, nor the specific range of backbone models. These lie beyond the abstract's information scope; readers who need to assess the robustness of the conclusions should consult the original paper and the released datasets.