Giving ESM3 a water attention layer: thermophile classification accuracy holds, but hydrogen bonds stop reading as destabilizing

The residue-level outputs of the structure-based protein language model ESM3 carry a systematic physical bias: it treats hydrogen bonds as destabilizing, and its prediction error grows with solvent-accessible surface area, indicating that missing explicit solvent modeling introduces bias at the protein–water interface. Earlier structure-based protein models treated proteins as if in vacuum, modeling neither solvent molecules nor solvent–protein codynamics; ESM3's coordinate-informed encoding supports geometry awareness but lacks force-field awareness. On a thermophile classification task built from 5125 homologous sequences, hydrogen bonds were systematically identified as destabilizing regardless of intermediate dimension, contradicting established biophysics and an RMSF benchmark built from 70 proteins; below 96 intermediate dimensions, glycine was also misidentified as stabilizing.

HEIWA injects solvent-mediated force-field information into ESM3's static residue embeddings by letting every protein query attend to all water keys, rather than assigning coordinates to individual solvent molecules. The module uses three water key types for hydrogen, lone pair and oxygen, with a sparsity constraint preventing the hydrogen-bond donor and acceptor heads from attending to the same residue; ReLU replaces softmax so one water head can act on many surface residues at once. Of 120 hyperparameter configurations, four with four solvent heads passed the biophysical screen and showed improved correlation of per-residue outputs with RMSF and SASA; replacing HEIWA with Gaussian random vectors left no configuration passing the screen.

Wired into the ESM3 generation pipeline, HEIWA yields generated proteins with somewhat greater kinetic stability at high temperature while largely preserving the overall structure of the ESM3 embedding. ESM3-generated proteins had been reported as apparently unstable, and the absence of solvent encoding may be related. Across 19 proteins shorter than 256 amino acids and two random seeds, 38 systems were run through 100 ns temperature ramping from 310 to 368 K; paired Wilcoxon tests found 23 proteins more stable with full HEIWA, 7 similar and 8 less stable, with an unsigned binomial test p = 0.011 after excluding ties, while the reversed-SASA-gate version showed no significant difference.

The demonstrated scope is ESM3 residue embeddings on a thermophile classification proxy task, plus generated structures for 19 monomeric non-membrane proteins shorter than 256 amino acids under a 310 to 368 K temperature-ramping protocol; the authors propose combining all-atom representations with solvent encoding as a next direction.

The authors note that weak interactions such as π–π stacking may be masked by solvent encoding, and that the low frequency of aromatic residues leaves it unclear whether this reflects missing force-field information or the small data set; HEIWA's effect on the RMSF of disordered regions was inconsistent across random seeds and structures, which the authors say likely needs greater statistical power; metadynamics free-energy profiles serve only as qualitative context and are not used for quantitative ranking.