# Recurrent Depth

A vision Transformer's depth-wise stack is compressed into a single recurrent Transformer block, where the FFN at each recurrent depth is a convex combination of a small shared expert bank programmed by a normalized depth coordinate, so shared weights still yield depth-dependent computation. Under supervised ImageNet-1k training, reViT-B/16 reaches DeiT III accuracy with about 70% fewer stored parameters; under distillation that matches only the DINOv2 teacher's final-layer features, an 8-expert model approaches the teacher on frozen classification, segmentation, and depth probes. One checkpoint also runs at multiple tested depths by resampling the depth coordinate.

A vision Transformer's depth-wise stack is compressed into a single recurrent Transformer block, where the FFN at each recurrent depth is a convex combination of a small shared expert bank programmed by a normalized depth coordinate, so shared weights still yield depth-dependent computation. Under supervised ImageNet-1k training, reViT-B/16 reaches DeiT III accuracy with about 70% fewer stored parameters; under distillation that matches only the DINOv2 teacher's final-layer features, an 8-expert model approaches the teacher on frozen classification, segmentation, and depth probes. One checkpoint also runs at multiple tested depths by resampling the depth coordinate.

The work measures how far pretrained transformers follow in-context reference chains: thirteen base models reliably follow only 1.4–3.6 lines (median 2.2) and extra pretrained loops add little, yet with all model weights frozen a task-trained rank-8 LoRA at one early layer extends this computation, raising Qwen3-8B from 15.5% to 99% exact accuracy on 24-line chains, reaching 50 lines with a longer-trained LoRA and, in Ouro-1.4B, 60 lines after four loops and at least 160 after eight, via a relay in the middle layers where frozen heads read progressively further up the chain and removing parent-line attention stops it.

The work measures how far pretrained transformers follow in-context reference chains: thirteen base models reliably follow only 1.4–3.6 lines (median 2.2) and extra pretrained loops add little, yet with all model weights frozen a task-trained rank-8 LoRA at one early layer extends this computation, raising Qwen3-8B from 15.5% to 99% exact accuracy on 24-line chains, reaching 50 lines with a longer-trained LoRA and, in Ouro-1.4B, 60 lines after four loops and at least 160 after eight, via a relay in the middle layers where frozen heads read progressively further up the chain and removing parent-line attention stops it.

Through controlled pretraining experiments, this study systematically examines when recurrence helps, where it should be applied, and how it should be conditioned in looped language models (LoopLMs), finding that extra loops beyond the training horizon improve reasoning (e.g., a reasoning score rising from 28.52 to 31.49) while degrading knowledge (from 62.80 to 52.11), that non-recurrent output layers improve robustness to under-unrolling, and that the proposed history-state injection (especially channel-wise) combined with timestep conditioning better preserves knowledge under extended unrolling and improves robustness across inference budgets.

Through controlled pretraining experiments, this study systematically examines when recurrence helps, where it should be applied, and how it should be conditioned in looped language models (LoopLMs), finding that extra loops beyond the training horizon improve reasoning (e.g., a reasoning score rising from 28.52 to 31.49) while degrading knowledge (from 62.80 to 52.11), that non-recurrent output layers improve robustness to under-unrolling, and that the proposed history-state injection (especially channel-wise) combined with timestep conditioning better preserves knowledge under extended unrolling and improves robustness across inference budgets.