# Embedding Drift

Intermediate checkpoints of open-source models show that language abilities form in sequence: morphological structure begins before roughly 100M tokens, while semantic embeddings start only after 1B tokens and mature much later. The study evaluates Pythia and OLMo3 checkpoints under teacher forcing and probes divergence recognition and associative link generation with a modified divergent-association task and the Remote Associates Test. Larger training data helps divergence recognition, while co-occurrence in pre-training and repetition at inference weaken both abilities.

Intermediate checkpoints of open-source models show that language abilities form in sequence: morphological structure begins before roughly 100M tokens, while semantic embeddings start only after 1B tokens and mature much later. The study evaluates Pythia and OLMo3 checkpoints under teacher forcing and probes divergence recognition and associative link generation with a modified divergent-association task and the Remote Associates Test. Larger training data helps divergence recognition, while co-occurrence in pre-training and repetition at inference weaken both abilities.