Oral resistome benchmark flips its winner: Hyena leads on held-out clusters, but MoR holds up on external databases
Synopsis
Sequence models for oral resistome proteins lacked an evaluation that rules out homology-driven memorization. The study clustered 26,324 proteins into 1,437 clusters at 40% sequence identity, trained eight causal architectures under one protocol, and validated externally on CARD and NCBI AMRFinderPlus. Hyena reached the lowest held-out perplexity at 14.93, but only four of eight architectures beat the residue-composition baseline of 17.99 externally, where parameter-matched MoR led at 17.40 and the Transformer degraded by 27%.
Interpretation
On the oral resistome, the ranking of sequence models depends on how the evaluation partitions data, not on architecture alone. These protein operators had not been evaluated under conditions that expose memorization, even though resistance gene families are highly redundant and random splits leave close relatives of test proteins in training. After clustering at 40% identity, Hyena reached held-out perplexity 14.93±0.58 against 15.33 for the Transformer, but on the external CARD set the Transformer rose from 15.33 to 19.44, a 27% degradation, and only four of eight architectures fell below the residue-composition baseline of 17.99.
Recursive architectures hold up better on external proteins and cost less compute. The fixed-depth Transformer degraded by 27% under external validation, while constrained architectures using compression or weight sharing degraded less. Parameter-matched MoR scored perplexity 17.40±0.08 on CARD and 17.36 on NCBI AMRFinderPlus, lowest of the eight, and the 0.79M-parameter MoR reached 1294k tokens/s at 4096 residues, 5.7 times the Transformer's 225k.
Adaptive depth genuinely allocates recursion per residue rather than spending a uniform depth. A fixed-depth model applies the same number of block applications to every residue. Averaged over three seeds, the fraction of residues still active at recursion depths one through four was 100%, 70%, 42%, and 22%, with 2.34±0.15 block applications per residue, 42% fewer than the four a fixed-depth model would spend.
Perspective
The evaluation covers 26,324 resistance-associated proteins from oral genera curated from UniProtKB, CARD, and NCBI AMRFinderPlus; the models are small residue-level language models trained from scratch, and all evaluation is computational, with no phenotype or susceptibility testing.
The corpus covers only proteins that public databases already annotate as resistance-associated in oral genera, genus assignment is coarse and some well-represented genera are mostly non-oral, drug-class labels are a regular-expression approximation across two curation systems, the models are small and trained from scratch and are not comparable to large pretrained protein language models, and for proteins with no detectable homology to training none of the eight architectures beat residue composition.