Co-LMLM interleaves vector retrieval queries with next-token prediction during pre-training, reaching lower perplexity at 360M than models trained on 40x more data and SimpleQA performance in line with gpt-4o-mini and above Claude Sonnet 4.5
The work proposes continuous-query limited memory language models (Co-LMLM), an LLM that interleaves flexible vector retrieval queries with next-token predictions and is pre-trained to copy knowledge returned from the KB rather than memorize it, using a scalable approach that jointly trains a knowledge-externalizing LLM, induces its knowledge base, and learns an expressive continuous retrieval mechanism; across pre-training at multiple model scales, Co-LMLM outperforms prior knowledge-externalizing and vanilla LLMs in both perplexity and factual precision, and at 360M scale it achieves lower perplexity than models pre-trained on 40x more data and SimpleQA-verified performance in line with gpt-4o-mini and higher than Claude Sonnet 4.5.