Multimodal pretraining stops guessing: 268 runs yield a data-mixing law for text, vision and audio
A study of text, vision and audio mixture-of-experts pretraining fits a data-mixing scaling law from 268 runs and draws actionable conclusions: audio should spend compute on parameters, repeating vision or audio data beyond two passes buys little, and vision interferes most with audio.