Molecular completion gets a harder test: shuffle or swap the spectra, and see whether the model actually used them

Property-to-structure reasoning now has a controlled evaluation: a model sees a masked molecular scaffold plus one set of physical properties and picks the missing fragment from two to four candidates. Earlier evaluations centered on end-to-end molecular reconstruction, where a correct prediction does not necessarily mean the result is grounded in the supplied properties, since structural priors can carry the answer. The benchmark holds 136 task families and 394 distinct candidate structures, and its 11 property representations supply 1,496 source-property instances with matched shuffled controls and 2,766 eligible within-family counterfactual substitutions.

The evaluation splits property use into two diagnostics: reliance asks whether correctly paired properties raise selection accuracy over shuffled ones, and responsiveness asks whether the model switches to the matching counterfactual candidate after the properties are replaced. The primary Property-Grounded Recovery metric gives equal weight to source recovery and candidate-targeted counterfactual recovery, averaging counterfactual correctness within each family first so families with more candidates do not carry more weight. Seven models are evaluated under a closed protocol with no tools, retrieval, task-specific training, or solved few-shot examples, and only input-validated records enter scoring.

The ranking of property channels differs between the reliance and responsiveness diagnostics. Luna, Sol, and Astra all show a larger reliance margin for atomwise 1H than for 13C peaks, yet a lower counterfactual flip rate; for Sol the reliance margins are 25.2 versus 12.6 percentage points and the flip rates are 53.6% versus 67.3%. IR gives the largest or joint-largest reliance gain for six models, at 39.7, 49.3, and 52.2 percentage points for Luna, Sol, and Astra and 47.1 for Opus; under IR substitution the flip rate is 80.3% for Luna and 89.8% to 91.2% for Sol, Opus, and Astra.

The evaluation targets small-molecule completion with computed properties and candidates that were curated and filtered by channel-specific separation thresholds; models receive only the masked scaffold, allowed attachment fields, candidate SMILES, and the selected property information, not the candidates' full property table. Evaluation runs under a closed protocol with no tools, retrieval, task-specific training, or solved few-shot examples, across seven models spanning closed-weight and open-weight groups. The authors propose expanding molecular families and property channels through LLM-assisted dataset construction while retaining expert review and validation.

The benchmark covers small-molecule completion with computed properties and curated candidates; experimental spectra and open-ended structure elucidation remain outside its scope, and current coverage is constrained by the effort required for careful dataset construction and manual validation. NMR observations expose only nuclei retained on the visible scaffold, so the masked group's own signals are unavailable and the model must infer the completion from effects on the remaining nuclei. Case studies show that all four closed-weight models err more often when molecular dipoles contradict a local polarity heuristic; the authors note that these substitutions change electronic structure and conformation together, so the contrast identifies a limitation of the heuristic rather than geometry as its isolated cause.