VeRL-Omni v0.3.0 brings async execution to multimodal RL training: short answers still favor sync, long ones favor separate-async
Synopsis
VeRL-Omni v0.3.0 extends asynchronous execution to multimodal reinforcement-learning training across text, audio input and diffusion models, and makes the V1 trainer built on TransferQueue and ReplayBuffer the default. Previously a strictly alternating loop left generation and training idle in turn. In MiniCPM-o 4.5 audio-visual question answering, separate-async ran about 9% slower per prompt on natural short answers but about 6.7% faster per prompt at forced 1024-token responses, so the execution mode should follow the workload.
Interpretation
Multimodal reinforcement-learning training can now choose synchronous or asynchronous execution within one stack, coordinating generation, reward scoring, policy updates and weight synchronization in a single pipeline. Previously a strictly alternating loop left one role idle while another worked, even though autoregressive generation, diffusion denoising and audio-video reward decoding each need different resources. MiniCPM-o 4.5 GSPO audio-visual question answering training gives two paths: separate-async scores 0.764 at step 13 versus 0.762 for colocated sync, and validation reward at step 10 is 0.747 for both.
The choice of execution mode is made a measurable configuration rather than a one-size-fits-all prescription. Separate-async places the FSDP actor and the vLLM-Omni rollout engine on different GPU pools, connects generation and training through a replay buffer, and synchronizes policy at configurable intervals. On a single node of 4 × H800 80 GB, colocated 4+0 (2×TP2) at batch size 128 versus separate-async 3+1 (1×TP1) at batch size 120 gives 1.522 versus 1.663 seconds per prompt at about 101 natural tokens, and 2.813 versus 2.624 seconds at 1024 tokens.
The diffusion side also gains asynchronous capability, including weight synchronization, dynamic hybrid rollout switching, and asynchronous teacher scheduling with standalone teacher routing for diffusion on-policy distillation. When the replay buffer runs low, colocated actor replicas can be lent to rollout and reclaimed for training on demand. The MiniMax H3 DiffusionNFT audio-video post-training case uses 8 GPUs, 16 samples per prompt, LoRA rank 64, and CLAP plus ImageBind rewards; the reported training reward mean rises from about 0.27 to close to 0.4 over roughly 60 steps.
Perspective
The demonstrated scope is the recipes and case studies in the VeRL-Omni v0.3.0 release: MiniMax H3 DiffusionNFT audio-video post-training, and the MiniCPM-o 4.5 GSPO audio-visual question answering comparison of colocated sync and separate-async, the latter on a single node of 4 × H800 80 GB at constant learning rate with forced response lengths. Readers should treat it as measured evidence for choosing an execution mode, not as a cross-version performance ranking.
The reported training reward rise shows only that the selected reward signal is being optimized and is not by itself evidence of a general improvement in visual quality or temporal consistency; the qualitative examples are silent previews and cannot show audio quality or synchronization. The timing figures are steady-state means over 20 consecutive steps of a single run, not independent trials, and the two arms use different batch sizes, so the per-prompt view is the one to compare. The H3 DiffusionNFT launchers still run on the legacy V0 path, so these numbers should not be read as V1 results.