Language skills do not grow together: morphology starts before 100M tokens, semantics only after 1B

A language model's linguistic abilities do not take shape together: morphological structure both begins and matures before semantic embeddings do. Most earlier work on language acquisition examined fully trained models, leaving the developmental trajectory across intermediate training stages largely unexplored. Across Pythia scales and OLMo3 checkpoints, the first turning point for morphological preference appears before about 100M tokens, while semantic onset comes after 1B tokens; Pythia's morphological plateau falls between 1B and 10B tokens, and its semantic plateau only after 100B tokens.

Morphological acquisition follows three phases, initialization, accelerated learning and a maturity plateau, and embedding drift traces the same three-part structure. Semantic feature evolution had been described as initialization, emergence and convergence, but whether morphological competence shares that rhythm had not been compared across time and scale. With teacher forcing computing generation probabilities for paired grammatical and ungrammatical targets, Pythia prefers favorable morphological forms by a factor of roughly 30 to 35 after the second turning point, and OLMo3 shows the same trend.

Divergence recognition and associative link generation depend jointly on training data volume, co-occurrence and repetition rather than improving monotonically with training. No earlier study had tracked how creativity measures evolve during language acquisition, or how other linguistic competencies might affect them. On 16 divergent-association word lists, both the initial development ratio and the final maturity ratio sit near the share of the most divergent triplets among all 120 triplets; OLMo3, which consumes about 19 times more training tokens, shows growing divergence recognition, while the two Pythia models do not.

The conclusions rest on publicly available intermediate checkpoints of the Pythia suite and OLMo3, evaluated through paired-target probabilities under teacher forcing, with morphology covering limited unit tests over three lexical categories, verbs, nouns and pronouns, where pronouns cover possessive and gender agreement. The authors note that the teacher-forcing setup cannot probe lexical categories that open the incipit prompt, such as personal pronouns. Divergence recognition uses a recall-based modification of the divergent-association task, and association generation uses the Remote Associates Test, with 16 mid-range ten-word lists drawn from a public dataset. The work addresses researchers interested in training dynamics and language-ability evaluation, and its conclusions apply to the studied open model families and their training data.

Morphological testing is limited to simple unit tests over the defined lexical categories; the authors note that a comprehensive test of morphological learning is beyond their scope and that the teacher-forcing setup cannot probe categories such as personal pronouns that open the incipit prompt. OLMo3 has not released early checkpoints near the first turning point, so the onset of its morphological acquisition is not directly observed. For divergence recognition, the initial development ratio and final maturity ratio sit near the triplet share, which the authors suggest may reflect a weakness of the recall-based task relative to the Remote Associates Test. On association generation, models favor surface thematic associations and frequent co-occurrence words, and the authors list expanding the morphological dataset, analyzing initialization effects and adding checkpoints as future directions.