LLMs read the elements but not the atomic arrangement: a geometric wall still separates them from materials property prediction

On data that pairs coordinates with categories, large language models use coordinate information far less effectively than category information. Earlier studies applied large language models to chemical and materials property prediction and reported encouraging results, which are hard to reconcile with documented failure modes. On synthetic data whose coordinate contribution can be tuned continuously, the coordinate contribution loss of language models is consistently larger than the category contribution loss, and this gap appears across datasets with different coordinate geometries and category distributions.

The models attend to category cues such as element types rather than to the relative positions of atoms in space. A graph model based on COGN shows the opposite slope between coordinate and category contributions, and the descriptor-based MODNet sits near zero, so neither carries this bias. Across all datasets, n-gram models closely follow the behavior of language models fine-tuned on category alone or on category plus coordinates, and attention analysis shows coordinate information is not used during prediction.

In real materials property prediction, adding geometric information to the text does not necessarily improve performance, and scaling does not close the gap to geometric architectures. Scaling models and data had been treated as a general route to better performance and formalized in empirical scaling laws. Across more than 2,000 language model training jobs under fivefold cross-validation, dataset scaling changes root mean square error by less than 20%, model scaling is similarly modest for shear and bulk modulus, and language-model approaches sit at the highest-error end of the MatBench leaderboard.

The report covers three properties, shear modulus, bulk modulus and perovskite formation energy, from a limited set of benchmark datasets that may not represent the full diversity of materials property prediction. The architectures evaluated are encoder-based BERT and decoder-based Llama-2; multimodal models that fuse text with explicit coordinate embeddings, and architectures with geometric priors built into tokenization, were not investigated. The scaling study stops at 70 billion parameters using parameter-efficient fine-tuning.

Whether full fine-tuning or pretraining from scratch on coordinate-rich corpora would alter the observed scaling behavior remains an open question. Applying the coordinate–category framework to molecular property prediction and to biological sequence–structure tasks would test whether the gap generalizes beyond crystalline materials. Coordinate-aware tokenization schemes and causal interventions such as activation patching are proposed to test whether the limitation is intrinsic to transformer architectures or specific to their text-based interface.