Scaling laws bend at finite budgets: spectral and data heterogeneity make the compute-optimal exponent drift with scale
Synopsis
Classical neural scaling laws write excess risk as a power law in capacity and data and assume the compute-optimal allocation exponent stays fixed across scales. This analysis uses Mellin–Barnes contour integration to split finite-scale departures along the capacity axis into a three-term expansion, reduces multi-domain mixing to scalar water-filling via KKT duality, and shows the allocation exponent drifts in proportion to capacity and data exponent variances. Across 45 operator spectra, Pythia, Chinchilla's 245 runs and RegMix's 17 domains, the estimators give holdout error and cut high-compute allocation regret.
Interpretation
Learning curves at finite budgets are no longer treated as a straight power law; departures along the capacity axis become a three-term expansion in which spectral-geometric and regularization-filter terms pull the local slope in opposite directions. The classical form treats excess risk as a separable, scale-invariant power law and therefore predicts log-risk decays linearly and the optimal model-data trade-off is a straight line in log coordinates. On 45 spectral configurations, a 36-cell factorial grid over filter exponent, source decay and manifold dimension plus 9 pole-collision and gradient-flow slices, normalizing the local slope deviation by its analytic peak collapses all four systems onto one single-parameter master curve to machine precision.
The optimal multi-domain mixture reduces to a one-dimensional root-finding problem, and training keeps tilting quota at the margin toward slower-learning bottleneck domains. In practice domain mixtures are chosen by fixed heuristics or small-scale proxy regression, yet when domains have different sample exponents any static mixture becomes suboptimal as the total budget grows. On three- and four-domain mixtures, the share ratio normalized by each domain's crossover budget falls on an exact inverse law; at large budgets the residual fast-domain mass stays inside the analytic sandwich band and the closed-form approximation converges at a quadratic rate.
The compute-optimal allocation exponent is not constant; it drifts with the difference between capacity-side and data-side exponent variances, and frozen hyperparameters add an upward bias to the estimate. Reported compute-optimal allocation exponents range from the Kaplan et al. value to the Chinchilla value, and an analytic framework separating genuine scale drift from frozen-hyperparameter bias was missing. On Chinchilla's 245 runs, fitting on 221 small-scale pilot runs and evaluating on 24 held-out large models, trimming high-loss short-horizon runs lowers the estimated exponent and further reduces holdout MAPE below the raw power law.
Perspective
The analysis targets separable capacity and data risk, algebraic spectral decay and strict convexity along iso-compute slices; empirical validation is limited to Pythia's Standard and Deduped Pile, Chinchilla's 245 runs and RegMix's 17 training domains, and the Chinchilla portion uses digitized data that record only triples without per-run learning-rate schedules.
The Chinchilla data lack per-run learning-rate schedules, so high-loss short-horizon runs can only be filtered empirically by a threshold; the ill-conditioning and rank deficiency of multi-power fits on narrow pilot spans are structural, and extrapolation accuracy still depends on pilot span and noise level.