# ParaTTT

Test-time training layers replace the recurrent state with a small neural network trained online by gradient descent, which is more expressive than a linear memory but forces sequential computation because each token's gradient depends on every earlier update. ParaTTT instead iteratively refines an estimate of the hidden-layer pre-activation gradients, with each refinement built from a forward, a readout and a backward chunkwise-parallel pass, and the exact online recurrence is a fixed point of that iteration. At 4K context, two refinements reach the error of TTT-MLP's block approximation in less time, and 340M and 1.3B models match or exceed Gated DeltaNet and Mamba-2 on perplexity and recall.

Test-time training layers replace the recurrent state with a small neural network trained online by gradient descent, which is more expressive than a linear memory but forces sequential computation because each token's gradient depends on every earlier update. ParaTTT instead iteratively refines an estimate of the hidden-layer pre-activation gradients, with each refinement built from a forward, a readout and a backward chunkwise-parallel pass, and the exact online recurrence is a fixed point of that iteration. At 4K context, two refinements reach the error of TTT-MLP's block approximation in less time, and 340M and 1.3B models match or exceed Gated DeltaNet and Mamba-2 on perplexity and recall.