# Genomic Language Models

The Allen Institute for Brain Science and AWS have released Bloom, a single-cell multimodal model that encodes gene expression and chromatin accessibility into one shared cell representation while training on roughly 100-fold fewer cells than comparable pretrained models. On adult mouse whole-brain data, linear classifiers on Bloom embeddings reach 91% cross-validated adjusted balanced accuracy for 315 subclasses (multiome), 94% for 310 subclasses (scRNA-seq) and 94% for 297 subclasses (snRNA-seq), although the model saw only mouse multiome data.

The Allen Institute for Brain Science and AWS have released Bloom, a single-cell multimodal model that encodes gene expression and chromatin accessibility into one shared cell representation while training on roughly 100-fold fewer cells than comparable pretrained models. On adult mouse whole-brain data, linear classifiers on Bloom embeddings reach 91% cross-validated adjusted balanced accuracy for 315 subclasses (multiome), 94% for 310 subclasses (scRNA-seq) and 94% for 297 subclasses (snRNA-seq), although the model saw only mouse multiome data.

The authors introduce RAGenome, the first retrieval-based genomic language model: it retrieves homologous sequences from a 100-vertebrate whole-genome alignment, discards the 73% of tokens that are gaps, and scales pretraining context to 13,312 nucleotides, 100 times longer than existing MSA-based gLMs; it raises gene-finding MCC from GPN-Star's 0.45 to 0.60 and reaches 0.87 AUROC on pathogenic-variant prioritization, unifying long-range and evolutionary signals with 168M parameters and 16,032 GPU hours.

The authors introduce RAGenome, the first retrieval-based genomic language model: it retrieves homologous sequences from a 100-vertebrate whole-genome alignment, discards the 73% of tokens that are gaps, and scales pretraining context to 13,312 nucleotides, 100 times longer than existing MSA-based gLMs; it raises gene-finding MCC from GPN-Star's 0.45 to 0.60 and reaches 0.87 AUROC on pathogenic-variant prioritization, unifying long-range and evolutionary signals with 168M parameters and 16,032 GPU hours.

This study maps the in-context learning operating regime of Evo 2, a nucleotide-level foundation genomic language model, across five binary classification tasks spanning biological and artificial sequences, finding robust performance on shorter natural sequences (F1=0.902 for miRNA, 0.785 for Toxins), degradation with sequence length and collapse at kilobase scale, no benefit from model scaling (the 7B model systematically outperforms the 40B variant), poor prediction of accuracy by perplexity, and mechanistic interpretability via logit-lens and Jacobian Scope suggesting a prediction-generalisation trade-off and that models might track prompt structure rather than signal-carrying content.

This study maps the in-context learning operating regime of Evo 2, a nucleotide-level foundation genomic language model, across five binary classification tasks spanning biological and artificial sequences, finding robust performance on shorter natural sequences (F1=0.902 for miRNA, 0.785 for Toxins), degradation with sequence length and collapse at kilobase scale, no benefit from model scaling (the 7B model systematically outperforms the 40B variant), poor prediction of accuracy by perplexity, and mechanistic interpretability via logit-lens and Jacobian Scope suggesting a prediction-generalisation trade-off and that models might track prompt structure rather than signal-carrying content.