Multimodal pretraining stops guessing: 268 runs yield a data-mixing law for text, vision and audio

Data mixtures for multimodal pretraining now have an extrapolatable quantitative law instead of trial and error. Earlier public scaling recipes covered a single modality or two modalities, and lacked a unified framework handling text, vision and audio together with repetition and data imbalance. The law was tested on three held-out runs; eight of nine modality-level predictions matched observed validation loss within 2.6%, six within 1.1%, with a correlation of 0.9.

The law splits each modality's validation loss into irreducible loss, effective parameters and effective data, where effective parameters decay exponentially with the non-target mixture mass and effective data discounts repetition while crediting cross-modal transfer. The standard Chinchilla form writes loss as a power law of parameter count and token count alone, so it cannot express help or interference between modalities. Fitting used mixture-of-experts models from 457M to 8.3B parameters, up to 150B tokens per run, totaling roughly 300k GH200-hours.

Cross-modal effects are asymmetric: vision interferes most with audio, text and vision interfere negatively with each other, and audio's transfer to text and vision has standard deviations exceeding its means in leave-one-scale-out folds, so it is treated as nearly inert. Earlier bimodal interaction laws assumed symmetric transfer and only balanced mixtures. At a mixture where the target modality holds 95% of training tokens, effective parameter retention is about 0.95 for text, 0.90 for vision and only 0.75 for audio.

The law targets native multimodal mixture-of-experts pretraining, for comparing modality mixtures under a given compute and data budget, and was checked on three held-out runs: a 12.6B-A3.8B from-scratch run and two 8.3B-A2.6B continual-pretraining runs. The authors point future work to substantially larger scales (hundreds of billions of active parameters, trillions of tokens), additional modalities such as long-form video, and a mechanistic account of why audio interferes so little.

The authors state that fitted coefficients depend on tokenizer choice, the next-token-prediction objective, MoE architectural hyperparameters and data quality, so the result is best viewed as a framework for fitting context-specific multimodal scaling laws rather than a universal law prescribing exact mixtures; the audio-source transfer coefficients have standard deviations exceeding their means across leave-one-scale-out and random-subset refits, and their sign is unstable. Modeling bidirectional capacity sharing did not consistently improve extrapolation, and the text loss error is larger in the non-text-heavy continual-pretraining run, which the authors attribute to insufficient text tokens for the loss to stabilize.