# Asynchronous Rollout

2 related articles

Subscribe to updates

vLLM Blog Nov 6, 2026
VeRL-Omni v0.3.0 brings async execution to multimodal RL training: short answers still favor sync, long ones favor separate-async
VeRL-Omni v0.3.0 extends asynchronous execution to multimodal reinforcement-learning training across text, audio input and diffusion models, and makes the V1 trainer built on TransferQueue and ReplayBuffer the default. Previously a strictly alternating loop left generation and training idle in turn. In MiniCPM-o 4.5 audio-visual question answering, separate-async ran about 9% slower per prompt on natural short answers but about 6.7% faster per prompt at forced 1024-token responses, so the execution mode should follow the workload.
VeRL-Omni v0.3.0 extends asynchronous execution to multimodal reinforcement-learning training across text, audio input and diffusion models, and makes the V1 trainer built on TransferQueue and ReplayBuffer the default. Previously a strictly alternating loop left generation and training idle in turn. In MiniCPM-o 4.5 audio-visual question answering, separate-async ran about 9% slower per prompt on natural short answers but about 6.7% faster per prompt at forced 1024-token responses, so the execution mode should follow the workload.
VeRL-Omni Asynchronous Rollout DiffusionNFT Group Sequence Policy Optimization Post-training
arXiv Oct 14, 2026
Async RL squeezes context into fewer decode steps; INT4 KV cache speeds rollouts but loses accuracy, bounded INT8 protection recovers it
Sustained concurrency in asynchronous RLVR rollout concentrates context processing into fewer decode steps, making the KV cache rather than weights the dominant cost. KV4-RL stores the full context in calibration-free INT4 and protects only attention sinks and recent context in bounded INT8; across full-parameter GRPO training on Qwen3-4B-Base and Qwen3-8B-Base, its six-benchmark mean accuracy stays within 0.5 percentage points of BF16 rollout while substantially accelerating rollout and full iterations.
Sustained concurrency in asynchronous RLVR rollout concentrates context processing into fewer decode steps, making the KV cache rather than weights the dominant cost. KV4-RL stores the full context in calibration-free INT4 and protects only attention sinks and recent context in bounded INT8; across full-parameter GRPO training on Qwen3-4B-Base and Qwen3-8B-Base, its six-benchmark mean accuracy stays within 0.5 percentage points of BF16 rollout while substantially accelerating rollout and full iterations.
Sihwan Park（First author）
KV4-RL KV Cache Quantization Asynchronous Rollout Reinforcement Learning with Verifiable Rewards GRPO

You’re all caught up

VeRL-Omni v0.3.0 brings async execution to multimodal RL training: short answers still favor sync, long ones favor separate-async

Async RL squeezes context into fewer decode steps; INT4 KV cache speeds rollouts but loses accuracy, bounded INT8 protection recovers it