# Word Sense Disambiguation

During pretraining of Pythia language models, word-sense disambiguation improves in sharp steps around 1,000 to 2,000 training steps, two layer-3 attention heads in 14M track that improvement and their ablation measurably hurts it, while 410M spreads the function across more heads that hold up better under position and part-of-speech perturbations.

During pretraining of Pythia language models, word-sense disambiguation improves in sharp steps around 1,000 to 2,000 training steps, two layer-3 attention heads in 14M track that improvement and their ablation measurably hurts it, while 410M spreads the function across more heads that hold up better under position and part-of-speech perturbations.

The paper proposes a context-aware semantic similarity (CASS) method for unsupervised word sense disambiguation: given a target word, its context and an exclusion list, the context and the context with each candidate synonym substituted are encoded as vectors, and the synonym that minimizes the resulting semantic change (maximizes cosine similarity) is chosen as the sense decision; evaluated for accuracy on CoarseWSD-20, a Wikipedia-derived, noun-only benchmark of 20 words with 2 to 5 senses and 10,196 instances, against a random-option baseline (43.73%) and a most-frequent-sense baseline (73.43%), the BERT-embedding configuration reached 77.74% (7,927 hits), the only setting above the strong baseline, while USE (71.94%), ELMo (68.75%) and WMD (60.

The paper proposes a context-aware semantic similarity (CASS) method for unsupervised word sense disambiguation: given a target word, its context and an exclusion list, the context and the context with each candidate synonym substituted are encoded as vectors, and the synonym that minimizes the resulting semantic change (maximizes cosine similarity) is chosen as the sense decision; evaluated for accuracy on CoarseWSD-20, a Wikipedia-derived, noun-only benchmark of 20 words with 2 to 5 senses and 10,196 instances, against a random-option baseline (43.73%) and a most-frequent-sense baseline (73.43%), the BERT-embedding configuration reached 77.74% (7,927 hits), the only setting above the strong baseline, while USE (71.94%), ELMo (68.75%) and WMD (60.