FactorEngram factorizes n-gram memory over a shared dictionary, lifting 8K long-context retrieval by up to 43.3 points at 340M and 1B backbones
The work proposes FactorEngram, which replaces monolithic per-pattern n-gram embeddings with sparsity-regularized coefficients over a dictionary shared across patterns and reuses that same dictionary for basis-level contextual gating, improving language modeling, downstream tasks, and long-context retrieval on 340M- and 1B-parameter Transformer backbones.