Public articles linked to the same research event.
arXiv Revisiting residual connections from an optimization perspective, this work proves that the layout of residual connections can fundamentally shape convergence behavior and even induce an exponential gap in convergence rates, and introduces ANCRe, a principled and lightweight framework that parameterizes and learns residual connectivities from data; ANCRe adaptively reassigns residual connections with negligible computational and memory overhead (<1%) to use network depth more effectively, and extensive numerical tests across large language model pretraining, diffusion models, and deep ResNets show consistently accelerated convergence, boosted performance, and enhanced depth efficiency over conventional residual connections.
Revisiting residual connections from an optimization perspective, this work proves that the layout of residual connections can fundamentally shape convergence behavior and even induce an exponential gap in convergence rates, and introduces ANCRe, a principled and lightweight framework that parameterizes and learns residual connectivities from data; ANCRe adaptively reassigns residual connections with negligible computational and memory overhead (<1%) to use network depth more effectively, and extensive numerical tests across large language model pretraining, diffusion models, and deep ResNets show consistently accelerated convergence, boosted performance, and enhanced depth efficiency over conventional residual connections.
Revisiting residual connections from an optimization perspective, this work proves that the layout of residual connections can fundamentally shape convergence behavior and even induce an exponential gap in convergence rates, and introduces ANCRe, a principled and lightweight framework that parameterizes and learns residual connectivities from data; ANCRe adaptively reassigns residual connections with negligible computational and memory overhead (<1%) to use network depth more effectively, and extensive numerical tests across large language model pretraining, diffusion models, and deep ResNets show consistently accelerated convergence, boosted performance, and enhanced depth efficiency over conventional residual connections.
Revisiting residual connections from an optimization perspective, this work proves that the layout of residual connections can fundamentally shape convergence behavior and even induce an exponential gap in convergence rates, and introduces ANCRe, a principled and lightweight framework that parameterizes and learns residual connectivities from data; ANCRe adaptively reassigns residual connections with negligible computational and memory overhead (<1%) to use network depth more effectively, and extensive numerical tests across large language model pretraining, diffusion models, and deep ResNets show consistently accelerated convergence, boosted performance, and enhanced depth efficiency over conventional residual connections.