LLMs learn which elements are present but not where atoms sit, so geometric models keep winning on materials properties
Synopsis
On coordinate–category data, language models learn which elements are present but not how atoms are arranged. Using synthetic datasets with tunable coordinate and category weights, plus the MatText benchmark across many text representations and scales, coordinate-dominated tasks show consistently higher error than category-dominated ones, a coordinate–category cliff. Scaling to about two million structures and 70 billion parameters does not remove it, while graph neural networks and other geometric architectures stay ahead on structure-dominated properties.
Interpretation
On data that pair coordinates with categories, language models use coordinate information far less effectively than category information, a gap quantified as the coordinate–category cliff. Materials property prediction had long relied on hand-crafted features or graph neural networks, and researchers recently turned to large language models as a general-purpose alternative, reporting encouraging results. On synthetic datasets whose coordinate and category contributions can be tuned continuously, error on the coordinate-dominated side stays above the category-dominated side across datasets with different coordinate configurations and category distributions.
The gap comes from model architecture and inductive bias rather than from the task itself. Running the same data through different architectures reverses how error moves with the coordinate weight. A COGN-based graph model handles coordinate-dominated tasks comparatively more easily, the descriptor-based MODNet sits near zero with balanced sensitivity to both information types, and language models show the opposite slope.
Text representation, dataset size and model size are each insufficient to recover coordinate information for language models. Scaling is usually treated as a reliable route to better performance, an expectation established in general language modeling and in chemistry. Within the MatText framework, growing data from 30,000 to about two million structures and models from 7 billion to 70 billion parameters changes root mean square error mostly by less than 20%, and absolute performance stays far below graph neural network approaches.
Perspective
The conclusions rest on three material properties, shear modulus, bulk modulus and perovskite formation energy, drawn from a limited set of benchmark datasets that may not represent the full diversity of property prediction tasks. The transformers evaluated are encoder-based BERT and decoder-based Llama-2; multimodal models fusing text with explicit coordinate embeddings, and architectures with geometric priors built into tokenization, were not investigated. The scaling study stops at 70 billion parameters using parameter-efficient fine-tuning.
Whether full fine-tuning, or pretraining from scratch on coordinate-rich corpora, would alter the observed scaling behavior remains an open question. The hypothetical potential in the synthetic data simplifies real interactions, and whether the coordinate–category gap generalizes beyond crystalline materials to other physical sciences domains still needs testing.