A model splits a word's two meanings early in pretraining: two heads carry 14M, while 410M spreads the job across steadier ones
During pretraining of Pythia language models, word-sense disambiguation improves in sharp steps around 1,000 to 2,000 training steps, two layer-3 attention heads in 14M track that improvement and their ablation measurably hurts it, while 410M spreads the function across more heads that hold up better under position and part-of-speech perturbations.