# Generalization Bounds

3 related articles

Subscribe to updates

arXiv Nov 4, 2026
Chain-of-thought solves unseen multi-step tasks by reusing one step predictor, with error growing sublinearly in depth
Chain-of-thought out-of-distribution generalization is framed as compositional generalization: training supplies only single-step atomic transitions, while the target task requires composing many steps. Theory bounds target risk by the sum of step risks times a depth-independent coefficient, and shows that sharing one predictor makes the estimation term shrink as the inverse square root of depth rather than grow linearly. Atom-Task and GSM8K experiments support short-to-long and unseen-combination transfer, and show explicit state tracking is essential.
Chain-of-thought out-of-distribution generalization is framed as compositional generalization: training supplies only single-step atomic transitions, while the target task requires composing many steps. Theory bounds target risk by the sum of step risks times a depth-independent coefficient, and shows that sharing one predictor makes the estimation term shrink as the inverse square root of depth rather than grow linearly. Atom-Task and GSM8K experiments support short-to-long and unseen-combination transfer, and show explicit state tracking is essential.
Nanjing University Singapore Management University
Nuojing Liang（First author）
Chain-of-Thought Out-of-Distribution Generalization Generalization Bounds Compositional Generalization Next-Token Prediction
arXiv Oct 29, 2026
More Communication Can Widen the Generalization Gap in Streaming Federated Learning, Bounds Grow With √R
Streaming federated learning trains a shared model from continuously arriving client data, whose streams are often temporally dependent. For clients whose data follow Markov processes, an analysis uses information-theoretic and rate-distortion tools to bound the expected generalization gap of the aggregated model, specializing to linear and self-attention next-token prediction. The bounds show that more communication rounds loosen the guarantee at roughly √(R/N), while temporal dependence enters through the square root of the mixing time; GPT-style pretraining on TinyStories and ETTh2 forecasting both show train and test losses falling with rounds while the gap widens.
Streaming federated learning trains a shared model from continuously arriving client data, whose streams are often temporally dependent. For clients whose data follow Markov processes, an analysis uses information-theoretic and rate-distortion tools to bound the expected generalization gap of the aggregated model, specializing to linear and self-attention next-token prediction. The bounds show that more communication rounds loosen the guarantee at roughly √(R/N), while temporal dependence enters through the square root of the mixing time; GPT-style pretraining on TinyStories and ETTh2 forecasting both show train and test losses falling with rounds while the gap widens.
Université Gustave Eiffel
Masoud Kavian（First author）
Streaming Federated Learning Generalization Bounds Markovian Data Next-Token Prediction Self-Attention
arXiv Oct 5, 2026
Generalization error of Transformer neural quantum states is proven to decay inversely with in-context examples and depth, with required depth scaling only linearly in system size
This work develops a theoretical framework for the generalization behavior of Transformer-based neural quantum states under in-context learning (ICL): it proves that there exists a Transformer architecture whose pointwise prediction error (MSE) decreases inversely with both the number of in-context examples and the Transformer depth, and that the depth required to achieve this guarantee scales only linearly with system size (the number of particles in continuous systems or qudits in discrete systems); the authors further extend the analysis to full quantum states formulated as rank-one density operators, deriving MSE-based generalization bounds under physical constraints over both continuous and discrete domains, and corroborate the predicted scaling with numerical simulations.
This work develops a theoretical framework for the generalization behavior of Transformer-based neural quantum states under in-context learning (ICL): it proves that there exists a Transformer architecture whose pointwise prediction error (MSE) decreases inversely with both the number of in-context examples and the Transformer depth, and that the depth required to achieve this guarantee scales only linearly with system size (the number of particles in continuous systems or qudits in discrete systems); the authors further extend the analysis to full quantum states formulated as rank-one density operators, deriving MSE-based generalization bounds under physical constraints over both continuous and discrete domains, and corroborate the predicted scaling with numerical simulations.
Zhen Qin（First author）
Neural Quantum States In-Context Learning Generalization Bounds Transformer

You’re all caught up

Chain-of-thought solves unseen multi-step tasks by reusing one step predictor, with error growing sublinearly in depth

More Communication Can Widen the Generalization Gap in Streaming Federated Learning, Bounds Grow With √R

Generalization error of Transformer neural quantum states is proven to decay inversely with in-context examples and depth, with required depth scaling only linearly in system size