Rename every amino acid and ESM-2's contact map nearly vanishes: protein language models lean on residue identity, not coevolutionary statistics
Synopsis
A global one-to-one renaming of the amino-acid alphabet leaves alignment-level statistics, column entropies and pairwise edit distances unchanged, yet contact maps from ESM-2, MSA Transformer and AMPLIFY collapse, while an inverse-covariance Potts model stays bit-identical. Across nine proteins and three seeds, ESM-2 650M keeps an overlap of only 0.015 with its own original contact set after random relabeling of all twenty residues, versus 0.655 when relabeling stays inside BLOSUM62 similarity groups. The contact readout therefore depends on canonical residue identity and cannot be cited as proof that these models have internalized coevolutionary statistics.
Interpretation
Protein language model contact maps stop being preserved once the amino-acid alphabet is globally renamed, while a classical coevolutionary model returns identical contacts on the same input. High contact accuracy had been read as evidence that these models implicitly learn coevolutionary statistics and extract the same signal as alignment-based methods such as Potts. Averaged over nine proteins and three seeds, ESM-2 650M's overlap with its own original contact set falls to 0.015 under random relabeling of all twenty residues; MSA Transformer falls to 0.147 on the same relabeled alignments, while inverse-covariance Potts returns overlap 1.000 in all 136 cells.
The collapse depends on which letters are installed, not only on how many positions change. Relabeling alters many positions, so degradation could come from increased edit distance alone. On cytochrome c, in-class relabeling and both matched substitutions change the same 76 of 104 positions; in-class relabeling keeps overlap 0.699, while uniform random and composition-matched substitutions both give 0.013.
Scaling the model does not remove the effect and widens the gap between in-class and random relabeling. Language-model loss improves predictably with scale, which invites the expectation that the contact readout moves closer to coevolutionary inference. Across five ESM-2 sizes from 8M to 3B, random-relabeling overlap moves from 0.042 to 0.035, staying near zero, while in-class overlap rises with scale.
Perspective
The experiments cover nine proteins, four Pfam families, three readouts (categorical-Jacobian contacts, ESM-2's own supervised contact head and ProteinGym variant effects) and three protein language model families, all on short single-domain proteins; multi-domain proteins, shallow families, and models retrained or fine-tuned on relabeled corpora were not tested.
Invariance under relabeling is a necessary but not sufficient condition for a contact readout to be a pure function of column statistics; the interventions isolate alphabet anchoring without identifying its cause, and how much in-class relabeling protects the map varies by protein, with the in-class and random arms both near chance at m=20 on protein G B1.