Recurrent memory splits each write into two calls: whether to store a token, then how many slots to touch, lifting sparse-fact accuracy from 0.048 to 0.222
Synopsis
Recurrent-memory models compress history into a fixed set of slots, and earlier multi-slot memories wrote every token to the same number of slots, merging whether to store a token with how many slots to update. Factorized Write Routing separates the two: a learned gate first decides whether the current token is written, then a slot router chooses a fixed or token-adaptive number of slots, while queries still read all slots. Across ten seeds on two synthetic long-context tasks with four slots per write, the result reverses with information density: always writing reaches 0.472 long-context accuracy on dense Feature Recall versus 0.337 for the learned gate, while the learned gate reaches 0.222 on sparse-fact Passkey versus 0.048 for always writing.
Interpretation
Memory writing in a recurrent model becomes two separately controllable decisions: whether the current token enters memory, and how many memory slots it updates when it does. Earlier multi-slot memories wrote every token to the same number of slots, implicitly assuming every token deserves a write and the same update budget, so storage and allocation were bundled together. On the official RAVEN backbone, Raven-Fixed writing each token to two slots and Raven-Dense writing all slots were compared over paired seeds 1 to 10; on dense Feature Recall the long-context mean accuracy was 0.2882 versus 0.3772, a paired difference of 0.0890 whose 95% confidence interval excludes zero.
Which write policy wins flips with the information density of the context, and learned selective writing is clearly better on the sparse-fact task. In matched contrasts at a fixed four-slot write, the learned gate was worse than always writing on dense Feature Recall and better than always writing on sparse-fact Passkey. Over ten seeds at four slots per write, Feature Recall long-context accuracy was 0.472 for always writing versus 0.337 for the learned gate, and Passkey was 0.222 for the learned gate versus 0.048 for always writing, with both paired intervals excluding zero.
The write gate first makes a binary decision per token, the slot router then allocates slots only for written tokens, and the read side is not restricted by that choice. The gate trains its non-differentiable threshold decision with a straight-through estimator, and slot allocation can be a fixed width or a width set by accumulating routing probability to a mass threshold. Query attention normalizes over all memory slots, so a query need not know which slots held a fact; training supervises the gate and router only indirectly through answer cross-entropy at query positions.
Perspective
The conclusions rest on two procedurally generated key-value retrieval tasks: in Feature Recall nearly every context token may be queried, while in Passkey a quarter of context tokens are facts and the rest are distractors. The model has two layers, hidden dimension 256 and about 1.91M parameters, trained at lengths 32 and 128 and tested out to 512 and 1024. The authors propose next testing a controlled density sweep and natural text.
The two tasks differ in more than information density, so the results do not identify density as the sole causal variable; the adaptive-versus-fixed contrasts use unequal realized write budgets, so they reject a universal advantage for that adaptive rule but do not prove fixed allocation is intrinsically better. The diagnostic oracle gate uses fact labels unavailable at deployment and is not an upper bound; memory updates before reading the current token, so query tokens themselves can alter the state and part of the oracle gap may reflect query handling rather than context filtering. The implementation still passes a dense routing tensor to the recurrent kernel, so sparsity is logical rather than realized acceleration.