Inside a protein language model, binary patterns form a jellyfish: hydrophobic content sets the main axis while heptad phase curls into a ring
Synopsis
In ESM2 residue embeddings, polar/nonpolar binary patterns organize into a readable geometry. Generating 50 sequences for each of all 128 heptad binary patterns, 6400 in total, and running position-preserving PCA yields a jellyfish-shaped manifold: hydrophobic content spreads along PC1, while PC2 and PC3 encode heptad phase as near sine and cosine. A charge alphabet stays ordered, but scrambled amino-acid groupings or the non-protein model ByT5 lose the cyclic organization.
Interpretation
Inside a protein language model's embedding space, a sequence's polar/nonpolar arrangement compresses into a few interpretable directions rather than remaining an opaque high-dimensional vector. Earlier readings of such models mostly probed individual structural or functional properties, without an experimentally validated design system to calibrate the representation. Across 5900 binary-patterned sequences spanning 62 to 120 helical residues, the first principal component accounts for 18.6% of residue-embedding variance, correlates strongly with amino-acid hydrophobicity, and oscillates along the sequence with a period of about 3.6 residues per turn.
Extending the analysis to all 128 heptad binary patterns places them in a regular geometric arrangement within a shared representation space. Periodic signals within single patterns had been seen only for a few helix-compatible patterns, and how patterns relate to one another was unclear. A joint position-preserving PCA over 6400 sequences gives PC1 explaining 9.3% of total variance and separating sequences by hydrophobic content, with PC2 and PC3 at about 4.7% each, arranging cyclically permuted patterns into closed cycles that form the jellyfish shape.
The geometry depends on the chemical meaning of the amino acids, not on repetition alone. If the model performed only generic repeat detection, any binary grouping should produce the same ordered structure. A charge-based plus/minus alphabet still yields a similarly ordered manifold, whereas arbitrary amino-acid groupings strongly disrupt the organization; input-layer embeddings, randomly initialized ESM2 models and the non-protein language model ByT5 all lack the cyclic organization.
Perspective
The analysis covers final-layer residue embeddings of the 3-billion-parameter ESM2 model esm2_t36_3B_UR50D, on four-segment binary-patterned synthetic libraries, with structures predicted by ESMFold v1 using three recycling iterations; the conclusions are limited to this model, this sequence framework and this predictor.
The model did not distinguish successful from unsuccessful individual designs, ESMFold compactness predictions for the Kamtekar designs did not correlate with cooperativity or oligomerization, and the database homolog analysis cannot exclude a contribution from shorter shared motifs.