Looped reasoners stop keeping only the last output: attention residuals carry a 7M model to 24,960 layers with accuracy still rising

The state-passing rule of a looped reasoner changes: the next iteration receives a weighted sum over earlier outputs instead of only the most recent one. Under the carry-last rule, each block output went to the next iteration in full, so earlier computations survived only through the latest state and one noisy iteration could degrade later propagation. On Sudoku-Extreme, carry-last TRM peaks at 62.2% exact accuracy at 468 layers and falls to 58.1% at 1,872 layers, while the new connection keeps improving over the same depth and reaches 90.9%.

The weighting combines a content score with a temporal decay, and it can be written as a recurrence that keeps one weighted vector and one normalization scalar. Attending over the full history of outputs requires storing every step's representation, so memory grows with the iteration count. A shared pseudo-query scores each candidate once when it is produced, a learned exponential decay discounts older candidates, and the recurrence is algebraically identical to the full weighted sum, with per-token persistent state independent of the loop count.

The learned step size surges at the update that completes a puzzle and returns to baseline afterwards, while unsolved puzzles keep it suppressed. If the scorer gave every output equal weight, the step size would converge to the equal-weight baseline of 0.56. For puzzles solved at update 13 or later, the mean step size stays near 0.40 before solving, rises to 0.91 at the solving update, and returns to 0.56 within six updates; candidates decoding to a fully correct board have a median content score of 7.25 against an upper clipping bound of 8.0, and the score falls monotonically as wrong cells increase.

The report covers symbolic puzzles and abstract visual transformation tasks: Sudoku-Extreme, Maze-Hard, ARC-AGI-1 and ARC-AGI-2, with a 7M-parameter nested looped reasoner trained under the TRM deep-supervision procedure. The depth study runs without retraining on a fixed sample, from 156 to 24,960 layer evaluations. Persistent state is 582.8 KiB per puzzle, independent of the loop budget.

The authors note that the aggregated state lies in the convex hull of the candidate states and therefore does not amplify the norm, but a uniform bound over arbitrarily many iterations additionally requires the candidate norms to stay bounded. Ablations show that removing the content score lowers accuracy by 6 to 7 percentage points, and setting the score to zero, which reduces the connection to carry-last, drops accuracy to 1.5%, indicating the connection is tied to the state distribution seen in training. The depth study fixes the number of supervision segments and layers per segment and varies only the outer cycle count.