Reward-guided diffusion now reads value one step ahead: VaLoP cuts the high-noise gradient error and tops LIBERO and RoboCasa success
Synopsis
Reward guidance steers a frozen diffusion or flow model at sampling time by following the gradient of a value function, but both common estimates of that gradient, differentiating a one-step denoised estimate and a learned value model, are least accurate at high noise, where the coarse structure of a sample is decided. VaLoP evaluates the value at a less noisy lookahead state one sampler step ahead and pulls its gradient back with one vector-Jacobian product; across SD3.5-Medium, SD3.5-Large and FLUX.1-dev with three rewards, VaLoP and its amortized variant VaLoP-Fast raise the target reward above StitchVM and DPS, and VaLoP-Fast attains the highest success rates on LIBERO and RoboCasa.
Interpretation
Reward-guided diffusion gains an adjustable lookahead ratio: the value is no longer read only at the current noisy state, but first at a cleaner state one sampler step ahead, and its gradient is then pulled back to the current state. The two prior routes sat at opposite ends, DPS differentiating the reward of a one-step denoised estimate and learned value models such as StitchVM evaluating directly on noisy states, and both depart from the target value gradient at high noise. On a three-Gaussian two-dimensional mixture with a known exact value, DPS loses alignment with the exact value gradient as noise grows and StitchVM is least aligned at the highest noise, while the same value model is far more accurate at the cleaner lookahead state; on SD3.5, single guided steps with DPS and StitchVM raise HPSv2 less than half as much at high noise as at low noise.
Pulling the gradient back from the cleaner state requires an extra correction, the vector-Jacobian product of the generator velocity with that gradient, and the lookahead ratio places learned-value guidance and DPS at the two ends of one family. Lookahead alone reaches HPSv2 0.365 and PickScore 23.52, below the full VaLoP at 0.382 and 23.79, so the pullback correction carries most of the gain. In an SD3.5-Medium ablation with the HPSv2 reward, HPSv2 and PickScore peak at lookahead ratios from 0.4 to 0.6 and fall toward both ends; on robot policies, lookahead alone falls below the baselines on every LIBERO suite, and only adding the correction lifts it above them.
VaLoP-Fast amortizes the per-step generator backward pass with a small correction head, and for text-to-image models one head per generator is trained with the HPSv2 value and reused for all rewards. Full VaLoP needs one generator vector-Jacobian product per step, while VaLoP-Fast replaces it with one head forward pass and takes less time and memory than StitchVM in the same implementation on all three generators. On 200 DrawBench prompts with three shared seeds, VaLoP and VaLoP-Fast exceed StitchVM and pixel-space DPS on the target reward by more than one standard deviation in every reward and model setting, VaLoP-Fast exceeds VaLoP in three of nine settings, and on robots VaLoP-Fast is highest on every LIBERO suite, with 67.0% average LIBERO success and 47.4% on RoboCasa atomic tasks.
Perspective
The demonstrated scope is text-to-image guidance on SD3.5-Medium, SD3.5-Large and FLUX.1-dev, plus critic-guided robot policies in the LIBERO and RoboCasa simulators; all robot experiments run in simulation, and the authors state they will evaluate VaLoP and VaLoP-Fast on a real robot in future work.
The method requires a value model in the generator's latent space or a critic of clean actions, and VaLoP-Fast requires training a head once per generator or once per task; the error analysis bounds the average error of the value a value model regresses over generator states rather than the error of its gradient, so the gradient error need not decrease monotonically with the lookahead ratio; the lookahead state follows only when the sampler step is the exact flow map, which the step approximates.