Training algorithms stop needing a tuning sweep: on AlgoPerf v1's self-tuning track, Schedule-Free AdamW wins 13 submissions with one configuration

Training algorithms now have a comparison track that does not rely on per-task tuning: each submission supplies one hyperparameter configuration and must hit the target on nine workloads from different domains. The earlier AlgoPerf v0.5 ran on pre-A100 hardware with old budgets, and its external-tuning track allowed sampling hyperparameters per workload, while the self-tuning track used no hyperparameter sampling. Version 1 upgraded the fixed hardware to a 40 GB GPU configuration, recalibrated runtime budgets for that hardware, added a decoder-only language modeling workload on FineWeb-Edu 10B, and compared 13 PyTorch and JAX submissions under one ruleset.

The top-ranked entry is Schedule-Free AdamW v2 (PyTorch), which couples momentum interpolation with dynamic iterate averaging to remove the learning-rate decay schedule while maintaining batch normalization statistics. The legacy Schedule-Free AdamW submission lacked the batch normalization fix, so validation accuracy degraded on batch-normalization-dependent tasks. Version 2 runs the training forward-backward pass with strictly in-batch statistics and buffer updates disabled, then calibrates running statistics at the evaluation weights with an extra forward pass every third training step, turning ImageNet ResNet-50 from a failure mode into a stable run and reaching the validation target on six of the nine workloads.

This report is for researchers and engineers who choose or implement training optimizers, and it covers 13 PyTorch and JAX submissions on the AlgoPerf v1 self-tuning track, scored by wall-clock time-to-target on fixed hardware. It also reports step-efficiency, relaxed-target, hyperparameter saliency, and retention analyses, and notes that matrix-preconditioned and two-loop methods make stronger per-step progress but need systems optimizations such as parameter sharding to be competitive in runtime.

The report notes that in the step-efficiency analysis, submissions that did not reach the target within the wall-clock budget still receive an infinite score on those instances, and that an ideal setup would also have submission data up to some step budget. In the relaxed-target analysis, the same relative relaxation is not equally lenient across workloads; for example, the baseline NAdamW reaches the relaxed Criteo 1TB target in about 5% of the time it needs for the default target, compared with about 75% on FineWeb-Edu 10B. Step-time discrepancies between JAX and PyTorch still vary by workload, and why ResNet and ViT remain significantly JAX-biased is an ongoing investigation.