LLMs learn which elements are present but not where atoms sit, so geometric models keep winning on materials properties
On coordinate–category data, language models learn which elements are present but not how atoms are arranged. Using synthetic datasets with tunable coordinate and category weights, plus the MatText benchmark across many text representations and scales, coordinate-dominated tasks show consistently higher error than category-dominated ones, a coordinate–category cliff. Scaling to about two million structures and 70 billion parameters does not remove it, while graph neural networks and other geometric architectures stay ahead on structure-dominated properties.