Dory1.0-4B moves inference compute into a looped block: one checkpoint lifts its 12-benchmark average from 35.2 to 40.6
Synopsis
Dory1.0-4B is a 4-billion-parameter looped language model whose single checkpoint varies how many times a shared recurrent block runs at inference, letting one model serve different compute budgets. It was pretrained on 6 trillion tokens, the first 4.5 trillion in NVFP4, then extended to a 256K context and post-trained with SFT and RLVR. On the equal-weight average of 12 benchmarks it rises from 35.2 at one loop to 39.7 at two and 40.6 at four, and with two or more loops it beats Qwen3.5-9B, Gemma 4 12B and Gemma 4 26B-A4B on that average.
Interpretation
Dory1.0-4B lets one set of weights run its shared recurrent block more times on demand, spending extra computation in latent space rather than on longer outputs. Scaling test-time compute has mostly meant generating longer reasoning traces, so latency, KV-cache footprint and serving cost grow with every generated token. On the equal-weight average of 12 benchmarks it scores 35.2 at one loop, 39.7 at two and 40.6 at four, and with two or more loops it beats Qwen3.5-9B, Gemma 4 12B and Gemma 4 26B-A4B on that average.
The model splits its 72 layers into an input block that runs once, a 40-layer recurrent block that repeats, and an output block that runs once, with the input block's output re-injected before every loop. All recurrent-block parameters are shared across invocations, and the loop boundaries apply only per-token L2 normalization, adding no learned adapter parameters or outer residual. Pretraining first trained a non-looped model in NVFP4 for 4.5 trillion tokens, then introduced looping in BF16 over 1.5 trillion tokens, sampling loop counts from one to four and training with truncated backpropagation through loops.
A difficulty-aware loop curriculum in post-training routes lower-pass-rate prompts preferentially to deeper loops. The RLVR stage runs 200 steps of asynchronous GRPO across 23 environments, with rollout workers each fixed at a loop count. In an ablation the curriculum yields its largest gains at four loops, for example raising Terminal-Bench v2.1 accuracy from 14.0% to 21.4%.
Caching keys and values only from the last loop keeps attention KV memory at the one-loop size of a non-looped model for any loop count. With per-loop caching, the recurrent block's KV cache grows linearly with the loop count. At four loops, last-loop caching changes benchmark scores by roughly 0.1 to 2.0 points relative to per-loop caching, with mean output lengths changing little, from 1.0% fewer tokens on GPQA Diamond to 3.4% more on Terminal-Bench v2.1.
Perspective
The demonstrated scope is a 4-billion-parameter text model evaluated on 12 benchmarks spanning coding and agentic tasks, reasoning, instruction following, long context and general knowledge; evaluation uses a shared protocol, and some scores differ from externally published results.
Gains from looping are task-dependent, with AA-Omniscience accuracy essentially unchanged across loop counts; last-loop caching exposes earlier-loop queries to final-loop keys and values rather than the same-loop entries used in training; and self-speculative decoding gains generally decrease at larger batch sizes.