Looped language models gain reasoning but lose knowledge when unrolled beyond the training horizon, and history-state injection with timestep conditioning mitigates the trade-off
Through controlled pretraining experiments, this study systematically examines when recurrence helps, where it should be applied, and how it should be conditioned in looped language models (LoopLMs), finding that extra loops beyond the training horizon improve reasoning (e.g., a reasoning score rising from 28.52 to 31.49) while degrading knowledge (from 62.80 to 52.11), that non-recurrent output layers improve robustness to under-unrolling, and that the proposed history-state injection (especially channel-wise) combined with timestep conditioning better preserves knowledge under extended unrolling and improves robustness across inference budgets.