# Peak Masked Language Modeling

Single-cell gene-regulation modeling has split into two paths: sequence models read DNA regulatory grammar but lack cell-level resolution, while single-cell omics models treat genes and peaks as fixed-vocabulary tokens and discard the sequence. Muss derives peak and gene representations from DNA with a frozen sequence encoder, links them through peak–peak self-attention and distance-biased peak–gene interactions, and trains with peak masked language modeling plus paired scATAC-seq and scRNA-seq. Across four paired datasets it still predicts expression for genes and peaks unseen in task training, reaching pseudobulk PCC 0.684 on PBMC10K in the unseen-sequence setting.

Single-cell gene-regulation modeling has split into two paths: sequence models read DNA regulatory grammar but lack cell-level resolution, while single-cell omics models treat genes and peaks as fixed-vocabulary tokens and discard the sequence. Muss derives peak and gene representations from DNA with a frozen sequence encoder, links them through peak–peak self-attention and distance-biased peak–gene interactions, and trains with peak masked language modeling plus paired scATAC-seq and scRNA-seq. Across four paired datasets it still predicts expression for genes and peaks unseen in task training, reaching pseudobulk PCC 0.684 on PBMC10K in the unseen-sequence setting.