Skip to main content
Search

# Convergence Analysis

5 related articles

Subscribe to updates
  1. arXiv

    AdamW's fixed points match an implicit objective, but its own dynamics can leave them unstable

    AdamW's fixed points coincide with the stationary points of a constrained, regularized implicit objective, yet those points need not be stable under AdamW's own dynamics. Working in the continuous-time limit, the analysis gives exact one-dimensional stability conditions and sufficient higher-dimensional ones, with examples where even a strongly convex objective loses stability and, in the common momentum regime, trajectories oscillate persistently instead of converging. Rescaling weight decay by the strength of the gradient update yields AdamMe, which keeps the same fixed points and admits a Lyapunov function guaranteeing that bounded trajectories converge.

  2. arXiv

    Stochastic diffusion sampling drops the higher-order detour: F-DDPM evaluates at a cleaner lookahead point and lowers FID at matched NFE

    Stochastic diffusion samplers have lagged deterministic ones, and raising approximation order does not necessarily improve their strong convergence order. F-DDPM evaluates the data-prediction model at a cleaner lookahead point built from the previous prediction and the previously injected noise, still with one model evaluation per iteration and a first-order strong convergence guarantee. On CIFAR-10, ImageNet, FFHQ and LSUN it reaches lower FID than the evaluated stochastic baselines at matched NFE budgets, and with 1.5 times the NFE it approaches or beats deterministic baselines in many settings.

  3. arXiv

    Kernel and neural-network solvers for HJB equations on unbounded domains: convergence and error bounds from empirical residuals

    For two non-monotone, mesh-free approximations of Hamilton–Jacobi–Bellman equations on unbounded spatial domains—Wendland kernel collocation under a native space norm constraint and physics-informed neural networks with smooth activations under a Sobolev-norm constraint—the work derives error bounds from computable empirical squared residuals: norm constraints, an interpolation inequality and a controlled-diffusion truncation estimate convert the empirical loss into the pointwise residual control needed for viscosity-solution error estimates, yielding uniform convergence on compact sets plus a posteriori and high-probability quantitative bounds governed by the fill distance for kernels and by network and sample sizes for neural networks.

  4. arXiv

    DAFL's distributed augmentation allocation lifts minority-class F1 to 0.183 and shortens convergence on CIFAR-10 federated learning

    The work first links data augmentation to federated learning convergence behavior, showing that reducing label proportion imbalance accelerates convergence, and then proposes DAFL: with clients' class counts known, the server assigns the minimum augmentation per client-class pair by minimizing augmentation and training time while constraining global class imbalance; on six imbalanced configurations of CIFAR-10 with eight classes and eight clients, DAFL achieves higher minority-class F1 with shorter training time on D2, D4, D5, and D6, reaching minority-class F1 0.183±0.055 and convergence time 188.119±86.344 s on D2, outperforming FedProx and Focal Loss.

  5. arXiv

    Researchers derive architecture-aware convergence bounds for selective state space models and compare nine federated learning algorithms across six text domains

    Addressing the unclear behavior of modern selective state space models such as Mamba2 in distributed learning, the work derives architecture-aware gradient and smoothness bounds for single- and multi-layer selective SSMs and convergence bounds for FedAvg and FedProx, characterizing how recurrent stability, input-dependent discretization, and state projection norms affect federated optimization; it then numerically validates the single-layer bounds on sequences generated by a teacher SSM, uses the analysis to formulate expectations about local training and client heterogeneity, and examines these expectations by comparing nine federated learning algorithms on Mamba2 language modeling across six text domains.

You’re all caught up