Skip the full retrain: small-scale runs plus a shared scaling law can tell whether a recommendation feature is worth shipping
Synopsis
Every feature change in a recommendation system normally needs a full-scale retrain for A/B testing, which is expensive, while a single small-scale run often gives a misleading signal. Wukong, HSTU and LLM-based recommenders are shown to follow the same functional form of scaling law after different features are added, differing only in fitted coefficients. Fitting that law to a handful of small-scale runs and applying statistical inference predicts, with confidence intervals, whether a feature helps, is neutral or regresses at full scale, using a fraction of full-scale compute.
Interpretation
After a new feature is added, a recommendation model's performance still follows the same family of scaling curves as training scale grows, with different fitted coefficients; the property is called feature-invariant scaling. Scaling laws had mostly described how performance changes with model or data scale, while deciding whether a feature helps usually meant adding it to the baseline, retraining from scratch on the full data and comparing. On Wukong, adding a dropout layer, switching to the SGD optimizer and adding an attention layer each produced a scaling curve fit by a power law, with fit values of 0.9953, 0.9923 and 0.9876 against a baseline of 0.9949.
The method trains the experimental model with the new feature at several smaller training scales, records validation loss, and extrapolates to full scale. A single small-scale run is unreliable: adding movie genre information to an LLM-based recommender lowered HR@10 at small scale but clearly improved it at full scale. Across Wukong, HSTU and an LLM-based recommender, covering more than 40 architectural, optimization and data changes, the method reports full-scale predictions with confidence intervals and calls a feature effective when the posterior probability of beating the baseline loss by more than 1% exceeds 90%.
The decision rule filters out neutral or regressive feature changes without full-scale training and identifies several changes that improve HSTU's state-of-the-art performance on MovieLens-20M. Full-scale A/B testing retrains the whole model for every candidate feature, and industrial settings can have hundreds of candidates, making per-feature full validation hard to afford. On HSTU, Time Gap Bias raised HR@10 from the baseline 0.3567 to 0.3603, Dropout 0.1 to 0.3590 and Pop Debias to 0.3585; the cumulative cost of the small-scale runs is a fraction of one full-scale training run.
Perspective
The demonstrated scope is more than 40 architectural, optimization and data changes on three recommendation model families (Wukong, HSTU, and LLM-based recommenders built on Llama-3.1 and Qwen), with the decision threshold set at a full-scale loss more than 1% below baseline and a posterior probability above 90%; the authors propose Bayesian sampling such as MCMC when the scaling law is not a power law.
The authors report one counterexample: on Qwen-0.5B, adding movie genre information made the scaling curve depart from a power-law fit and noticeably degraded performance; they hypothesize that the feature increases context length by about 30% and that a small model lacks capacity to absorb it, and they concede such settings would hinder the method while expecting them to be uncommon.