Looped reasoning stops keeping only the latest state: attention residuals hold solutions at nearly 25,000 layers
Synopsis
Looped Transformers deepen reasoning by re-running a shared block at test time, but the usual carry-last rule lets later iterations overwrite correct intermediate results. InfiLoop replaces that rule with learned attention over recurrent outputs: a shared pseudo-query scores each candidate once, a learned exponential decay discounts older ones, and the aggregation has an exact streaming recurrence needing only a weighted-state vector and a normalization scalar per token. On Sudoku-Extreme a 7M-parameter model reaches 97.9% exact accuracy versus 94.2% for FPRM, and hits 90.9% at 1,872 layers where FPRM needs 3,744 layers for 90.1%.
Interpretation
The bottleneck in looped reasoning is located in how iterations replace state: passing each newest block output in full leaves earlier computation reachable only through the latest state, so one noisy iteration can degrade useful representations downstream. Earlier remedies used decay or layer scaling to control residual growth and update magnitude, but the next loop still received only the latest recurrent state. On Sudoku-Extreme, two checkpoints trained with the same loop schedule run for 72 outer updates: carry-last TRM peaks at 62.2% at 468 layers and falls to 58.1% at 1,872 layers, while InfiLoop keeps improving over the same depth to 90.9%.
The connection instead aggregates attention over recurrent outputs, letting the model decide which intermediate states to keep rather than assuming the newest output is best. Aggregation no longer relies on the latest state as the only channel; a running summary holds earlier outputs and the connection learns how much of each new output to add. A shared pseudo-query scores each candidate once when produced, and a learned exponential decay reduces older candidates' weights; the aggregation has an exact streaming recurrence keeping one weighted-state vector and one normalization scalar per token, with persistent memory independent of loop count.
The aggregation is equivalent to an adaptive residual update whose step size is the ratio of the new candidate's content weight to the retained historical mass, so low-scoring candidates are suppressed and high-scoring ones accepted. A fixed exponential average also damps updates but has no content score and cannot distinguish good candidates from bad ones. At the 1,872-layer Sudoku-Extreme setting, removing the content score during training drops accuracy from 90.9% to 84.1%, removing it at test time drops it to 84.8%, and reducing the connection to carry-last leaves 1.5%.
Perspective
The report covers symbolic puzzles (Sudoku-Extreme, Maze-Hard) and abstract visual transformation tasks (ARC-AGI-1, ARC-AGI-2), with a 7M-parameter nested recurrent reasoner trained under TRM's deep-supervision procedure. The depth study runs without retraining on a fixed sample of Sudoku-Extreme test puzzles, up to 24,960 layer evaluations. It suits researchers interested in test-time compute scaling and looped architectures.
The content-score and decay ablations are run at the 1,872-layer Sudoku-Extreme setting, and comparable ablations are not reported for the other benchmarks. The perturbation experiment injects a relative perturbation at update 36 and unrolls for another 36 updates. The aggregated state stays within the convex hull of the candidate representations, but a uniform bound over arbitrarily many iterations additionally requires the candidate norms to remain bounded.