mRNA foundation models get a new readout: averaging 5′ UTR, CDS and 3′ UTR separately lifts translation and stability prediction by 11–25%
Synopsis
mRNA foundation models usually average token embeddings over the whole transcript before fitting a linear probe, so signals from short regulatory regions are diluted by longer ones. A parameter-free region-aware readout instead averages within the 5′ UTR, coding sequence and 3′ UTR and concatenates the three means; across 20 frozen models it raises Pearson correlation on translation and stability tasks by 11–25% on average, with no model significantly worse on any regression task. On classification tasks AUPRC changes range from −3% to +11%.
Interpretation
With the same frozen models, the same forward pass and the same linear probe, replacing the whole-transcript mean vector with three concatenated regional means improves translation-efficiency and stability prediction. The earlier mRNABench evaluation averaged token embeddings into one transcript vector, where the 5′ UTR holds roughly 5–10% of the nucleotides and its regulatory signal is outweighed by longer regions. Across 20 models the region readout scores 0.054–0.092 higher in Pearson correlation than the global mean, or 11–25%; TE human and TE mouse improve significantly in 20/20 models, MRL in 19/20 and HL in 18/20, and no model is significantly worse on any regression column.
The method assigns each nucleotide vector from the frozen model to the 5′ UTR, coding sequence or 3′ UTR by annotation, averages within each region and concatenates the three means into a vector three times as wide for the same linear probe. The readout carries no trainable parameters and leaves pretrained weights untouched, using only the region coordinates the benchmark datasets already provide. Reproducing global-mean scores for the 19 models mRNABench reports gives a median absolute difference of 0.0009 over 380 model–dataset–target cells; the region readout is evaluated on the same models, tasks and ten splits.
The gain depends on the annotated region boundaries rather than on wider features, region length or GC content. Random boundaries drop HL, MRL, TE human and TE mouse by 0.053–0.085, with all 20 models falling; shifting both boundaries 100 nt toward the 5′ end lowers MRL by 0.041 in all 20 models. An equal-thirds readout trails the region readout by 0.017–0.054, and still by 0.018–0.054 when both are reduced to the model's hidden size by PCA; appending region lengths and GC leaves the region readout 0.021–0.055 above the nucleotide mean.
Perspective
The results rest on the mRNABench release 1.3 datasets, ten splits and 20 frozen models; only the pooling is replaced, with encoders and probe estimators unchanged, and the authors propose reporting both readouts as complementary evaluations.
The authors state the two readouts are complementary rather than region pooling being uniformly better: the global mean is more sensitive to synonymous codon recoding and closer to measured in-frame uAUG effects in the four MPRA libraries, and inserting four AREs or let-7 sites moves neither readout by more than 0.10 SD relative to shuffled controls.