Muchen Li

The work introduces Mixture of Memory Embeddings (MoME), which replaces each token's single memory row with multiple memory slots and uses a learned gate over the hidden state to sparsely choose which slots to read at each position, giving memory retrieval a context-aware form while keeping cheap token-indexed lookup; in controlled pretraining across nanochat, Llama-3/MobileLLM, and Qwen3 backbones, MoME improves over Value Embedding, Bigram, and STEM baselines in iso-parameter and iso-training-FLOP settings, shows a more favorable memory-size scaling trend at sub-billion scale, remains efficient in training and inference, and routing analyses on polysemous tokens suggest the same surface token is dispatched to distinct memory slots under different senses.

The work introduces Mixture of Memory Embeddings (MoME), which replaces each token's single memory row with multiple memory slots and uses a learned gate over the hidden state to sparsely choose which slots to read at each position, giving memory retrieval a context-aware form while keeping cheap token-indexed lookup; in controlled pretraining across nanochat, Llama-3/MobileLLM, and Qwen3 backbones, MoME improves over Value Embedding, Bigram, and STEM baselines in iso-parameter and iso-training-FLOP settings, shows a more favorable memory-size scaling trend at sub-billion scale, remains efficient in training and inference, and routing analyses on polysemous tokens suggest the same surface token is dispatched to distinct memory slots under different senses.