Pretrained models skip sparse pretraining: a token-indexed memory lifts HumanEval by 4.27 points while throughput stays within 1.5%

A pretrained dense model can gain parametric capacity during continued training through added static sparse memory, without pretraining a sparse architecture from scratch. Sparse upcycling had mainly targeted mixture-of-experts models, where cloned experts can remain functionally similar and limit the benefit of further expert scaling. On OLMo-2 1B, the ten-task average rises from 44.39 under continued dense training to 45.59, with HumanEval up 4.27 points, MBPP up 2.80, GSM8K up 2.27, and DROP up 1.65.

The new memory is added as a residual pathway on top of the existing feed-forward up-projection rather than replacing it, so pretrained computation is retained. STEM replaces the feed-forward up-projection with a token-indexed embedding table, and in a four-layer ablation it fails to match continued dense training even with additional warmup. The embedding tables are zero-initialized to restore the seed model's computation graph, and warmup freezes the up-projections so the new tables learn complementary signals; on OLMo-2 7B the average rises from 57.11 to 58.01, improving 9 of 10 evaluated tasks.

The reported setting is continued training on OLMo-2 1B and 7B backbones, with the main 1B comparison using roughly 100B warmup tokens followed by 100B mid-training tokens, and the 7B run using a reduced budget of about 30B warmup plus 70B mid-training tokens. Embedding tables sit in early decoder layers, capacity scales along augmented-layer count and embedding dimension, and the large tables stay in CPU memory with only needed rows moved to the GPU. The authors propose examining whether jointly scaling static memory with mixture-of-experts can push the performance-efficiency frontier further.

Removing all embedding tables drops the ten-task average from 45.59 to 32.17, showing the model relies on the learned memory after mid-training, but the fully ablated model is not equivalent to the original dense seed because the remaining weights co-adapted with static memory. The modest MMLU gain on 7B may reflect the shorter warmup, consistent with the 1B warmup ablation, although that explanation is not isolated in the 7B experiments. Token-level interventions change outputs, and the authors state this does not establish general edit reliability or an isolated fact representation.