Skip to main content

# Implicit Bias

2 related articles

Subscribe to updates

arXiv Oct 29, 2026
Gradient descent's implicit bias is no longer set by initialization alone: step size tilts the limit toward a perturbed weighted L1 minimizer
The training limit of deep linear diagonal networks had been characterized only for gradient flow, the continuous version with vanishing step size, where the implicit bias is fixed entirely by initialization. This analysis rewrites discrete gradient descent and stochastic gradient descent themselves as perturbed mirror gradient descent, proves convergence of the iterates for any number of layers, and characterizes the limit as a convex minimization problem parameterized by a perturbed initialization. The perturbation shrinks linearly with step size, so the limit corresponds to weighted L1 minimization whose weights depend on initialization, step size and the whole iterate trajectory.
The training limit of deep linear diagonal networks had been characterized only for gradient flow, the continuous version with vanishing step size, where the implicit bias is fixed entirely by initialization. This analysis rewrites discrete gradient descent and stochastic gradient descent themselves as perturbed mirror gradient descent, proves convergence of the iterates for any number of layers, and characterizes the limit as a convex minimization problem parameterized by a perturbed initialization. The perturbation shrinks linearly with step size, so the limit corresponds to weighted L1 minimization whose weights depend on initialization, step size and the whole iterate trajectory.
Ludwig-Maximilians-Universität München
Wiebke Bartolomaeus（First author）
Implicit Bias Linear Diagonal Networks Gradient Descent Stochastic Gradient Descent Mirror Gradient Descent
arXiv Oct 7, 2026
Implicit Bias in Hyperbolic Multiclass Learning Is Governed by Busemann Risk: A Drift Coefficient Decides Whether Embeddings Escape to the Ideal Boundary or Return Inward
Studying the implicit bias of Riemannian gradient flow for hyperbolic multiclass classification with fixed prototypes, this work decomposes large-radius distances into a radial term and a Busemann direction term, proving a radial dichotomy in which the sign of a drift coefficient decides whether the radius escapes to the ideal boundary or returns inward, and showing that boundary directions converge to critical points of the Busemann risk, with an asymptotic classifier and decision boundaries on the ideal boundary.
Studying the implicit bias of Riemannian gradient flow for hyperbolic multiclass classification with fixed prototypes, this work decomposes large-radius distances into a radial term and a Busemann direction term, proving a radial dichotomy in which the sign of a drift coefficient decides whether the radius escapes to the ideal boundary or returns inward, and showing that boundary directions converge to critical points of the Busemann risk, with an asymptotic classifier and decision boundaries on the ideal boundary.
The University of Tokyo
Xingrun Li（First author）
Implicit Bias Hyperbolic Representation Learning Riemannian Gradient Flow Busemann Risk PERM Losses

You’re all caught up