# Neural Tangent Kernel

Training five operator models on the one-dimensional discrete Poisson problem shows that error is not learned wavenumber by wavenumber but is retained along small-eigenvalue directions of the kernel that governs the training update. Sine-aligned spectral and linear Fourier neural operators give each wavenumber its own rate, so error collects at high wavenumbers; dense networks have eigendirections close to random in the sine basis, so error is nearly equal at every wavenumber and relative error grows only because the true solution spectrum decays. Removing the minibatch cross terms leaves the GELU dense network with under 0.1% of squared error below ηλ≈8×10⁻⁵ under either forcing.

Training five operator models on the one-dimensional discrete Poisson problem shows that error is not learned wavenumber by wavenumber but is retained along small-eigenvalue directions of the kernel that governs the training update. Sine-aligned spectral and linear Fourier neural operators give each wavenumber its own rate, so error collects at high wavenumbers; dense networks have eigendirections close to random in the sine basis, so error is nearly equal at every wavenumber and relative error grows only because the true solution spectrum decays. Removing the minibatch cross terms leaves the GELU dense network with under 0.1% of squared error below ηλ≈8×10⁻⁵ under either forcing.

Majority-vote test-time reinforcement learning makes each question lock onto one answer during training, often a wrong one on hard questions, and until now which answer that is became known only after training. The work treats this as a short race driven by sampling noise between the answers that are close at the start, and gives a closed-form probability for how the network kernel shifts the gap of the two leaders through updates of the trailing answers. In LoRA training on MMLU-Pro multiple-choice questions with three models from two families, predictions from a frozen-kernel chain computed before training forecast which answer each question locks into better than a logistic model refitted on the outcomes, using no fitted constants.

Majority-vote test-time reinforcement learning makes each question lock onto one answer during training, often a wrong one on hard questions, and until now which answer that is became known only after training. The work treats this as a short race driven by sampling noise between the answers that are close at the start, and gives a closed-form probability for how the network kernel shifts the gap of the two leaders through updates of the trailing answers. In LoRA training on MMLU-Pro multiple-choice questions with three models from two families, predictions from a frozen-kernel chain computed before training forecast which answer each question locks into better than a logistic model refitted on the outcomes, using no fitted constants.

High-pressure phase identification has relied on dynamic graph convolutional networks (DGCNNs) to classify each atom's local environment, which is costly to run. A Gaussian kernel model instead places a set of atom detectors around the origin, sums the isotropic Gaussian density contributed by every atom at each detector, and feeds those values to an ordinary neural network. On test sets with 16, 32 and 64 neighbors it reaches 78%, 86% and 90% accuracy, close to DGCNN's 75%, 85% and 92%, while predicting a full simulation in about 20-30 minutes versus about 5-6 hours for the comparison model on the same HPC environment.

High-pressure phase identification has relied on dynamic graph convolutional networks (DGCNNs) to classify each atom's local environment, which is costly to run. A Gaussian kernel model instead places a set of atom detectors around the origin, sums the isotropic Gaussian density contributed by every atom at each detector, and feeds those values to an ordinary neural network. On test sets with 16, 32 and 64 neighbors it reaches 78%, 86% and 90% accuracy, close to DGCNN's 75%, 85% and 92%, while predicting a full simulation in about 20-30 minutes versus about 5-6 hours for the comparison model on the same HPC environment.

For Muon under momentum accumulation and a tuned finite-step Newton–Schulz update, the work proves that full-batch training of a sufficiently wide two-layer ReLU network reaches any target empirical squared loss ε with high probability when the constant learning rate scales as (1-μ)√ε, with a hitting-time bound of O((1-μ)^-1ε^-1/2) and a sufficient width independent of both target accuracy and momentum.

For Muon under momentum accumulation and a tuned finite-step Newton–Schulz update, the work proves that full-batch training of a sufficiently wide two-layer ReLU network reaches any target empirical squared loss ε with high probability when the constant learning rate scales as (1-μ)√ε, with a hitting-time bound of O((1-μ)^-1ε^-1/2) and a sufficient width independent of both target accuracy and momentum.

Lu and Tsai propose a multilevel stochastic-gradient neural solver (MLSG) for second-kind boundary integral equations that represents the unknown boundary density with a neural network optimized by stochastic residual minimization over a hierarchy of successively refined Nyström discretizations, warm-starting each level with the previous level's network parameters; the algorithm avoids grid-transfer operators and hierarchical fast-summation machinery, relying instead on batched kernel evaluations and standard network forward and backward passes, and is demonstrated on Laplace/Poisson and Helmholtz problems in two and three dimensions plus an exterior Robin problem on a hypersurface in R⁴, under both parametric and signed-distance surface representations at up to million-scale discretizati

Lu and Tsai propose a multilevel stochastic-gradient neural solver (MLSG) for second-kind boundary integral equations that represents the unknown boundary density with a neural network optimized by stochastic residual minimization over a hierarchy of successively refined Nyström discretizations, warm-starting each level with the previous level's network parameters; the algorithm avoids grid-transfer operators and hierarchical fast-summation machinery, relying instead on batched kernel evaluations and standard network forward and backward passes, and is demonstrated on Laplace/Poisson and Helmholtz problems in two and three dimensions plus an exterior Robin problem on a hypersurface in R⁴, under both parametric and signed-distance surface representations at up to million-scale discretizati