# Gradient flow

EDP-PI-DOSnet embeds classical energy-stable schemes into a physics-informed operator-splitting network so that the learned discrete operators satisfy a discrete energy dissipation law. The earlier PI-DOSnet did not guarantee that its learned linear operator was symmetric negative semidefinite, energy could rise between steps, and long-time extrapolation needed two or three rounds of retraining. On the Allen–Cahn equation, both EDP-PI-DOSnet variants keep the modified energy non-increasing at every observed time level and give smaller long-time prediction errors without retraining.

EDP-PI-DOSnet embeds classical energy-stable schemes into a physics-informed operator-splitting network so that the learned discrete operators satisfy a discrete energy dissipation law. The earlier PI-DOSnet did not guarantee that its learned linear operator was symmetric negative semidefinite, energy could rise between steps, and long-time extrapolation needed two or three rounds of retraining. On the Allen–Cahn equation, both EDP-PI-DOSnet variants keep the modified energy non-increasing at every observed time level and give smaller long-time prediction errors without retraining.

This work studies the mechanism of Muon's spectral orthogonalization with a tractable factual-recall model: with S subjects and R relations and S>R, gradient flow learns relations before subjects with a learning-time ratio of Θ̃(√(S/R)), while spectral gradient flow reduces that ratio to Θ̃(1) and makes subject and relation errors decay as 1/(T log T) and exp(-poly(T)) respectively; gradient flow and spectral gradient flow are equivariant under orthogonal transformations of token embeddings whereas Sign gradient flow is not, so different orthonormal embeddings can yield no feature separation, a large separation phase, or even a reversed learning order.

This work studies the mechanism of Muon's spectral orthogonalization with a tractable factual-recall model: with S subjects and R relations and S>R, gradient flow learns relations before subjects with a learning-time ratio of Θ̃(√(S/R)), while spectral gradient flow reduces that ratio to Θ̃(1) and makes subject and relation errors decay as 1/(T log T) and exp(-poly(T)) respectively; gradient flow and spectral gradient flow are equivariant under orthogonal transformations of token embeddings whereas Sign gradient flow is not, so different orthonormal embeddings can yield no feature separation, a large separation phase, or even a reversed learning order.