# OLMo 2

Pretrained dense models can be expanded during continued training instead of pretraining a sparse architecture from scratch. STAPLE keeps the pretrained feed-forward up-projection and adds zero-initialized token-indexed embedding tables as a residual pathway, so the new parameters learn complementary token-specific information. On OLMo-2 1B it raises the ten-task average from 44.39 to 45.59 over continued dense training, with HumanEval up 4.27 points, MBPP 2.80, GSM8K 2.27 and DROP 1.65, and gains persist on OLMo-2 7B.

Pretrained dense models can be expanded during continued training instead of pretraining a sparse architecture from scratch. STAPLE keeps the pretrained feed-forward up-projection and adds zero-initialized token-indexed embedding tables as a residual pathway, so the new parameters learn complementary token-specific information. On OLMo-2 1B it raises the ten-task average from 44.39 to 45.59 over continued dense training, with HumanEval up 4.27 points, MBPP 2.80, GSM8K 2.27 and DROP 1.65, and gains persist on OLMo-2 7B.

The authors introduce LUMOS, a diagnostic framework that uses OLMo 2 with its fully transparent training corpus to move LLM parametric-knowledge analysis from output-only inference to tracing the causal chain from training-data exposure to behavioral output, finding that models internally encode rare facts with high separability (84%) yet express them behaviorally only 54% of the time (a retrieval gap that narrows with scale), while self-reflection is reliable on trained content (83%) but drops to random-baseline levels (49%) on unseen content, a collapse that persists under chain-of-thought prompting, which inflates confidence signals rather than improving calibration.

The authors introduce LUMOS, a diagnostic framework that uses OLMo 2 with its fully transparent training corpus to move LLM parametric-knowledge analysis from output-only inference to tracing the causal chain from training-data exposure to behavioral output, finding that models internally encode rare facts with high separability (84%) yet express them behaviorally only 54% of the time (a retrieval gap that narrows with scale), while self-reflection is reliable on trained content (83%) but drops to random-baseline levels (49%) on unseen content, a collapse that persists under chain-of-thought prompting, which inflates confidence signals rather than improving calibration.