Ewha Womans University

The authors introduce a diagnostic benchmark with four task families covering structural grounding, synthesis-condition verification, evidence-sufficiency verification, and MLIP-based computational verification, evaluate T-MOF-1-3 under closed-book, retrieval-enabled, and oracle-evidence settings to localize failures in knowledge access, evidence acquisition, and reasoning, and build MOF-Verify, a failure-aware agentic harness that substantially improves MOF hypothesis-verification performance over direct inference and retrieval-based baselines across multiple backbone LLMs, with benchmark datasets released.

The authors introduce a diagnostic benchmark with four task families covering structural grounding, synthesis-condition verification, evidence-sufficiency verification, and MLIP-based computational verification, evaluate T-MOF-1-3 under closed-book, retrieval-enabled, and oracle-evidence settings to localize failures in knowledge access, evidence acquisition, and reasoning, and build MOF-Verify, a failure-aware agentic harness that substantially improves MOF hypothesis-verification performance over direct inference and retrieval-based baselines across multiple backbone LLMs, with benchmark datasets released.

The authors introduce LUMOS, a diagnostic framework that uses OLMo 2 with its fully transparent training corpus to move LLM parametric-knowledge analysis from output-only inference to tracing the causal chain from training-data exposure to behavioral output, finding that models internally encode rare facts with high separability (84%) yet express them behaviorally only 54% of the time (a retrieval gap that narrows with scale), while self-reflection is reliable on trained content (83%) but drops to random-baseline levels (49%) on unseen content, a collapse that persists under chain-of-thought prompting, which inflates confidence signals rather than improving calibration.

The authors introduce LUMOS, a diagnostic framework that uses OLMo 2 with its fully transparent training corpus to move LLM parametric-knowledge analysis from output-only inference to tracing the causal chain from training-data exposure to behavioral output, finding that models internally encode rare facts with high separability (84%) yet express them behaviorally only 54% of the time (a retrieval gap that narrows with scale), while self-reflection is reliable on trained content (83%) but drops to random-baseline levels (49%) on unseen content, a collapse that persists under chain-of-thought prompting, which inflates confidence signals rather than improving calibration.