LLM pretraining optimizers stop orthogonalizing layer by layer: TUON stacks layers into a tensor and beats Muon and TEON on four model scales
Large language model pretraining optimizers are moving from handling each weight matrix separately to stacking the momenta of one module's layers into a third-order tensor. Layer-wise Muon ignores cross-layer correlation, while matricization-based tensor optimizers handle only two-layer stacks and QKV. TUON applies a unitary transform along the layer mode, polarizes each transformed slice, and inverts the transform; it lowers validation loss versus Muon and TEON on GPT-124M/350M and Qwen-0.6B/4B at comparable step time.