Pretrained models gain sparse capacity without retraining: a zero-initialized memory path lifts ten-task average by 1.20 points
Pretrained dense models can be expanded during continued training instead of pretraining a sparse architecture from scratch. STAPLE keeps the pretrained feed-forward up-projection and adds zero-initialized token-indexed embedding tables as a residual pathway, so the new parameters learn complementary token-specific information. On OLMo-2 1B it raises the ten-task average from 44.39 to 45.59 over continued dense training, with HumanEval up 4.27 points, MBPP 2.80, GSM8K 2.27 and DROP 1.65, and gains persist on OLMo-2 7B.