Same molecule, shuffled spectra: a benchmark asks whether models actually use the evidence

Evaluation of molecular structure prediction now asks whether the answer moves with the supplied physical properties, not only whether the final answer is right. Earlier evaluations centered on end-to-end molecular reconstruction, where a model can reach the correct structure through structural priors while the supplied property evidence plays little role, and aggregate accuracy cannot show which property channels are actually used. Each task is a masked scaffold with a candidate list, and the same task is presented with correctly paired properties, shuffled properties, and counterfactual properties taken from another candidate in the same family, so the difference across conditions indicates whether the evidence is used.

The benchmark scores property-grounded recovery as its primary metric and adds two diagnostics that separate static reliance from dynamic responsiveness. A single accuracy number conflates reliance on correct property pairing with responsiveness to counterfactual substitution, and the two capabilities rank property channels differently. Property-grounded recovery gives equal weight to source recovery and candidate-targeted counterfactual recovery, averaging within family first so families with more candidates do not dominate; reliance margin compares accuracy under correct pairing against a shuffled control, and counterfactual flip rate measures how often an initially correct choice switches to the corresponding counterfactual candidate.

The benchmark spans 11 property channels and 136 task families, with 1,496 source-property instances and 2,766 within-family counterfactual substitutions. Previously there was no evaluation set that aligned properties across channels on the same scaffold and candidate set and allowed direct property substitution. Candidate families come from scaffold matching, structurally distinct completions, and channel-specific separation thresholds, with properties drawn from aligned QM9, QM9-NMR, and QM9S records; under 1,000 subsamples of 100 families, the aggregate rankings for reliance margin and counterfactual flip rate are retained in 97.7% to 99.3% of draws.

All seven models benefit from some properties, but the size of the benefit and the responsiveness to property changes rank channels differently. Infrared gives the largest or joint-largest reliance margin for six models, while several NMR representations yield near-zero or negative gains. Luna, Sol, and Astra reach infrared reliance margins of 39.7, 49.3, and 52.2 percentage points, and Opus reaches 47.1; under infrared substitution, counterfactual flip rates are 80.3% for Luna and 89.8% to 91.2% for Sol, Opus, and Astra.

The benchmark targets small-molecule completion with computed dipole, HOMO-LUMO gap, UV-Vis, infrared, Raman, and 1H and 13C NMR shielding, over curated candidate sets, under a closed protocol with no tools, retrieval, task-specific training, or solved few-shot examples. It suits researchers and engineers working on molecular property reasoning evaluation, spectra-to-structure inference, and diagnostics of evidence use.

The benchmark covers computed properties and curated candidates, while experimental spectra and open-ended structure elucidation remain outside its scope, and coverage is constrained by the effort required for dataset construction and manual validation. NMR observations expose only nuclei on the retained scaffold, so signals from the masked group are unavailable and the model must infer the completion from effects on retained nuclei. Case comparisons show that all four closed-weight models err more often when molecular dipoles contradict a local carbonyl-based polarity heuristic, and those contrasts identify a limitation of the heuristic rather than geometry as its isolated cause.