DNA-LLM2Vec turns causal DNA decoders into bidirectional encoders, with frozen-probe gains that shift by task and backbone

Causal DNA decoders encode only one-directional context, while many genomic tasks need sequence on both sides of a locus at once; DNA-LLM2Vec turns such decoders into frozen bidirectional encoders whose representations a frozen linear probe can read directly. Bidirectional context has mainly come from masked-language-model encoders, which supervise only about 15% of masked positions and are costly to scale over long windows; causal decoders supervise every token and scale more easily to long sequences but keep directional representations. On the DNAGPT and HyenaDNA backbones, frozen linear-probe margins of each adapted variant over its backbone-matched causal baseline are compared across seven biological task families, and the gains vary by family and backbone with no single recipe dominating.

The conversion runs in three steps: remove the causal mask or merge the long convolution in both directions so each position can read both flanks, then apply masked next-token prediction, still predicting a masked token from the preceding position and reusing the pretrained prediction head. Adding a mask token for standard masked-language-model fine-tuning of a decoder creates a vocabulary mismatch, because the mask token never appeared during autoregressive pretraining; masked next-token prediction keeps the causal convention that the preceding position predicts the current token and avoids that mismatch. DNAGPT is adapted with LoRA on 1024-token hg38 windows and HyenaDNA is fully fine-tuned on 8192 bp nucleotide windows; the contrastive stage branches from the same stage-2 checkpoint into dropout, reverse-complement, crop and local-shift positive views.

On extended-context families, crop contrastive learning is the shared optimum for both backbones, lifting DNAGPT's six EPI tasks by 7.8 to 22.6 percentage points MCC and Virus-COVID by 15.2 points macro-F1. Short-context families respond according to backbone and tokenisation: on DNAGPT the reverse-complement view gives the strongest mean response for histone marks, local regulatory elements, TF-binding and promoter tasks, while every adapted DNAGPT variant falls below the causal baseline on splice. HyenaDNA also improves on all EPI tasks by 9.2 to 13.9 points MCC and Virus-COVID by 20.2 points macro-F1, but its length trend is weaker because several shorter histone, promoter and splice tasks also gain substantially.

The conclusions rest on one human-only DNAGPT scale and one HyenaDNA backbone, evaluated on every task of four public benchmark suites aggregated into seven biological task families, with frozen linear probing as the primary protocol and full fine-tuning as a supplementary analysis. Variant-effect prediction is run on HyenaDNA only, to keep reference and alternate windows aligned position-for-position at single-base resolution.

The pretraining-value and crop-start controls use a single seed and should be read as indicative of the main effects rather than as variance-quantified estimates. The GUE+ EPI benchmark is sequence-level rather than coordinate-resolved, and small families such as Virus-COVID and splice are descriptive observations requiring broader validation. Tokenisation, sequence operator and context length cannot be fully separated without matched controls, so these remain plausible contributors rather than a proven mechanism.