Memory retrieval stops reading only whole-sentence vectors: reusing a frozen encoder's token states lifts Qwen3-4B average Recall@10 from 62.5 to 75.7

Ranking for memory retrieval now draws on local matches between query tokens and memory tokens, not only on one vector per memory. Single-vector retrieval scored relevance through whole-memory similarity and did not explicitly match a query to the small part of a memory that carries the answer. On Qwen3-Embedding-4B, fusion without query adaptation raises the eight-benchmark average Recall@10/NDCG@10 from 62.5/48.0 to 73.8/59.9, while direct MaxSim on the same backbone reaches 67.8/54.3 and ColBERTv2 reaches 70.8/59.5.

Each query token retrieves a small set of matching tokens across all candidate memories, their scores are aggregated by source memory, and the result is fused with whole-memory similarity for ranking. The local and pooled scores are standardized per query and then combined with a weight, so a memory can still appear in the results even when none of its tokens is retrieved. Evaluation covers 5,873 queries across eight benchmarks, LoCoMo, LongMemEval, REALTALK, ESG-Reports, QASPER, PeerQA, CovidQA and MLDR, on eight frozen backbones spanning Qwen3-Embedding, Llama-based models, EmbeddingGemma-300M and Jina-v5-text-nano.

An optional query adapter trains only a low-rank residual while the encoder and the stored memory representations stay fixed. Adaptation acts only on the local query branch, leaving the pooled query embedding and the memory index unchanged, so the bank is built once and reused. On Qwen3-Embedding-4B, query adaptation lifts the average Recall@10/NDCG@10 further to 75.7/62.3; answer accuracy rises from 63.44% to 67.86% on LoCoMo and from 67.80% to 72.60% on LongMemEval.

The report targets researchers and engineers working on memory retrieval and long-context question answering; evaluation spans eight retrieval benchmarks and eight frozen backbones, with downstream answering by Qwen3-30B-A3B-Instruct and a fixed GPT-4o-mini judge, and the main comparison retrieves 10 memories. On compression, four-bit scalar quantization reduces storage by 81.4–85.9% relative to the uncompressed index, with mean Recall@10/NDCG@10 changing by +0.33/+0.19 points.

As the token retrieval budget grows from 16 to 128, each Recall curve on LongMemEval spans less than 1.6 percentage points and each NDCG curve less than 0.9, and on LoCoMo both metrics decline for training-free Qwen-4B once the budget passes 32. Query adaptation benefits vary across backbones: Llama-3B shows a small decrease, while EmbeddingGemma is unchanged after rounding. Gains over the original embedding hold at every retrieved-memory budget across both datasets and all eight backbones.