# Looped Language Models

Dory1.0-4B is a 4-billion-parameter looped language model whose single checkpoint varies how many times a shared recurrent block runs at inference, letting one model serve different compute budgets. It was pretrained on 6 trillion tokens, the first 4.5 trillion in NVFP4, then extended to a 256K context and post-trained with SFT and RLVR. On the equal-weight average of 12 benchmarks it rises from 35.2 at one loop to 39.7 at two and 40.6 at four, and with two or more loops it beats Qwen3.5-9B, Gemma 4 12B and Gemma 4 26B-A4B on that average.

Dory1.0-4B is a 4-billion-parameter looped language model whose single checkpoint varies how many times a shared recurrent block runs at inference, letting one model serve different compute budgets. It was pretrained on 6 trillion tokens, the first 4.5 trillion in NVFP4, then extended to a 256K context and post-trained with SFT and RLVR. On the equal-weight average of 12 benchmarks it rises from 35.2 at one loop to 39.7 at two and 40.6 at four, and with two or more loops it beats Qwen3.5-9B, Gemma 4 12B and Gemma 4 26B-A4B on that average.

Through controlled pretraining experiments, this study systematically examines when recurrence helps, where it should be applied, and how it should be conditioned in looped language models (LoopLMs), finding that extra loops beyond the training horizon improve reasoning (e.g., a reasoning score rising from 28.52 to 31.49) while degrading knowledge (from 62.80 to 52.11), that non-recurrent output layers improve robustness to under-unrolling, and that the proposed history-state injection (especially channel-wise) combined with timestep conditioning better preserves knowledge under extended unrolling and improves robustness across inference budgets.

Through controlled pretraining experiments, this study systematically examines when recurrence helps, where it should be applied, and how it should be conditioned in looped language models (LoopLMs), finding that extra loops beyond the training horizon improve reasoning (e.g., a reasoning score rising from 28.52 to 31.49) while degrading knowledge (from 62.80 to 52.11), that non-recurrent output layers improve robustness to under-unrolling, and that the proposed history-state injection (especially channel-wise) combined with timestep conditioning better preserves knowledge under extended unrolling and improves robustness across inference budgets.