Recurrent memory no longer needs training: frozen networks recall over 256 steps with half the trainable parameters
Lead
In partially observable reinforcement learning, freezing the recurrent memory module and training only the output and mixer layers still matches fully trained counterparts on delayed recall, discrete and continuous control, and pixel-based tasks, while cutting trainable parameters by more than half.
Story
The recurrent memory module in partially observable reinforcement learning can stay untrained, keeping its randomly initialized weights, and the agent still solves control tasks that require long-horizon memory. Recurrent agents previously trained the memory module jointly with the policy or value function, so the memory parameters were themselves part of the learning target. On the delayed-recall task, frozen LRU still reaches a 100% success rate at delays up to 256 steps, and frozen GRU matches trainable GRU with a 100% success rate at a delay of 128 steps.
The recipe stops the gradient on the recurrent unit's parameters, letting the fixed recurrent network act as a temporal feature extractor while only the nonlinear readout and feature mixer behind it are trained. This separation comes from reservoir computing, where a randomly initialized recurrent network stays frozen and only the output mapping is learned. Across 18 POPGym settings, frozen LRU trains only about 40% of total agent parameters and reaches a final evaluation return of 0.464 versus 0.458 for LRU, while frozen GRU trains about 30% of total parameters and reaches 88.70% of GRU's final return.
Frozen memory also works together with a learnable visual encoder on high-dimensional partially observable control tasks that take images as observations. Earlier work on frozen recurrent networks evaluated only a limited range of partially observable environments and did not cover high-dimensional pixel observations. On four POPGym Arcade tasks, frozen LRU reaches a final evaluation return of 0.694 versus 0.624 for LRU, a relative increase of 11.2% with lower variance, and frozen GRU reaches 0.643, retaining 95.2% of GRU's 0.676.
What to watch
The report covers the evaluated partially observable tasks: delayed recall, discrete control on POPGym, continuous control on MuJoCo Playground, and pixel control on POPGym Arcade, where the recurrent unit is frozen while the readout and feature mixer remain trainable. The authors propose broader ablations, additional benchmarks, and stronger baselines to better assess the generality of the findings.
The authors state that they do not conduct a large-scale, exhaustive ablation, that the evaluation covers a limited set of benchmarks, and that computational constraints restrict the range of baselines considered. The generalized reservoir computing analysis relies on specific assumptions and establishes its result under a relaxed echo state condition; random initialization, optimization dynamics, and finite-sample effects add uncertainty, so trained agents may not always satisfy the theoretical conditions exactly.