Video flow alignment drops the either-or: guidance stays in sampling, and CFG-free inference keeps the reward gains

Reward alignment for video flow models separates the sampling distribution from the optimization anchor: guidance is used only to generate rollouts, while the regression target stays anchored on the unguided conditional velocity field. Earlier regression-based alignment required sampling and anchor to share one velocity field, so rollouts had to stay unguided; restoring fidelity with guided rollouts while anchoring on that same guided field makes training diverge. On Wan2.1-1.3B and 14B at matched GPU hours, unguided sampling falls behind during cold start and does not close the gap after recovering, while the anchor-on-CFG variant decreases from the first checkpoint and diverges before roughly 300 GPU hours.

The resulting policy needs only a single forward branch at inference, and its conditional-to-unconditional velocity gap shifts only by an advantage-bounded increment. Guidance distillation regresses the guided field into the network and gives up inference-time guidance; here the centered advantage cancels the guidance offset in expectation, so the conditional-to-unconditional gap is no longer compounded round over round. Trained on Wan2.1-1.3B with the HPSv3 reward, the policy overtakes Flash-GRPO at roughly 400 GPU hours under CFG-free inference and converges to a higher reward, rises further when CFG is enabled, while the policy-gradient baselines stay essentially unchanged.

Each rollout is supervised at an independently drawn training timestep, so one prompt group can cover several timesteps. Single-step schemes tie the training timestep to the sampling discretization or share one timestep across the group. On Wan2.1-1.3B, independent timesteps beat a group-shared timestep under this objective but do worse under a GRPO objective, indicating the freedom depends on the point-wise regression target.

The report covers text-to-video alignment on Wan2.1-1.3B and 14B, trained on the DanceGRPO prompt set and evaluated on the same 300-prompt split used by Flash-GRPO, with VBench and motion quality reported on 1.3B only. It addresses researchers and engineers working on online reward alignment for video flow models, and the inference side supports both CFG-free single-branch and CFG-enabled modes. The source lists integrating off-policy or out-of-distribution terminal data as future work.

Optimization is bounded by the base model's own generation capability, so prompts the backbone fundamentally fails to render yield no informative reward gradient regardless of the guidance scale. The group-size trade-off is non-monotone: smaller values estimate too coarsely and saturate early, while larger values are more accurate per update but complete too few updates within the same GPU hours. The source notes that anchor invariance mathematically permits any behavior distribution, yet only CFG-guided rollouts from the EMA backbone are currently used.