RNA language models learn structure: structRFM pre-trains on 21 million sequence–structure pairs and matches AlphaFold3 after tertiary adaptation

RNA foundation models no longer learn from sequence alone: structRFM puts base-pairing structure into the pre-training objective, so one model yields sequence-level, nucleotide-level and pairwise features for broad downstream adaptation. Earlier RNA language models relied mainly on large-scale sequence data, while structural information often came from noisy annotations or task-specific objectives that could limit generalizability. The model ranks highly in zero-shot homology classification and establishes new benchmarks for secondary-structure prediction, indicating that structure-guided pre-training still holds on tasks it was not tuned for.

The mechanism adds a pair-matching operation to masked language modelling, so a masked base also consults its potential pairing partner instead of guessing from sequence context alone. To reduce predictor bias in structural annotation, training uses MUSES, an ensemble of secondary structures from multiple sources; a dynamic masking ratio balances structure-guided masking against nucleotide-level masking. Pre-training uses 21 million RNA sequence–structure pairs; rRNA and tRNA account for 66.3% and 16.7% of the data, and canonical pairs make up about half of all base-pairing interactions, including GC at 26.5%, AU at 14.4% and GU at 7.2%.

The gain from structure guidance extends beyond secondary structure: Zfold, the tertiary-structure adaptation, improves both three-dimensional models and the secondary structures extracted from them. On CASP15-natural, CASP16 and RNA-Puzzles, Zfold improves three-dimensional predictions and their extracted secondary structures, with performance comparable to AlphaFold3. Evaluation covers 8 independent RNA targets for CASP15-natural, 13 for CASP16 and 20 for RNA-Puzzles, compared with ϵRMSD, TM-score, GDT-TS and INF.

The demonstrated scope is performance on zero-shot homology classification, secondary-structure prediction benchmarks, the CASP15-natural, CASP16 and RNA-Puzzles tertiary-structure evaluations, and function-prediction tasks including IRES identification and ncRNA classification; the pre-training data, model weights and source code are openly available.

The authors report that structRFM generally performs better on natural targets than on synthetic targets containing dense long-range interactions, but sometimes fails to predict short local stems, producing large multiple loops; in tertiary-structure evaluation it does better on viral and aptamer targets and worse on multimer and bacterial targets.