Finer process rewards are not automatically better: decaying the weight lifts long-horizon trailer editing by 45.4%, while a fixed weight truncates 44.1% of rollouts
Synopsis
On four trailer-editing task types built from 49 real films and their released trailers, decaying an LLM-judged turn-level process reward to zero within the same GRPO setup gives a 45.4% relative gain over outcome-only GRPO on Fine Cut, while the fixed-weight variant scores below outcome-only GRPO on the three shorter task types and truncates 44.1% of rollouts. Process rewards do encourage visual inspection and reasoning early, but continued optimization drives redundant tool calls without a committed result; token-level on-policy self-distillation proves highly sensitive to the teacher's privileged information.
Interpretation
Credit-assignment granularity for multi-turn video-editing agents is compared under one training setup: trajectory-level, turn-level and token-level signals share the base model, training tasks and number of updates. Earlier video-production agents mostly wrap a model that is never updated, and segment-level credit is used in a single setting, so different granularities lacked a comparable control. The comparison trains Qwen3.5-9B for 50 updates on 716 training tasks and evaluates four times on 81 validation tasks from six held-out films, reporting the mean.
The benefit of turn-level process rewards depends on task length: on Fine Cut, which averages more than 60 turns, GRPO+PRM with a decayed weight raises the score from 0.201 under outcome-only GRPO to 0.293. Outcome-only GRPO raises the overall score from 0.284 to 0.413, but Fine Cut improves by only 0.050 while the other three task types each improve by at least 0.15. On the three shorter task types averaging 30 to 40 turns, the decayed weight yields at most a 10.4% relative gain over outcome-only GRPO, which remains competitive.
A fixed process-reward weight induces reward hacking: the process reward keeps rising while task completion and outcome scores decline, and the agent keeps calling tools without committing a result. Compared with outcome-only GRPO, the fixed-weight variant raises the truncated-rollout rate from 0.3% to 44.1% and scores lower on every task type except Fine Cut. Decaying the weight to zero between updates 20 and 30 lowers truncation to 6.2% and raises the overall score to 0.447, the highest among the tested schemes.
Token-level on-policy self-distillation does not reliably benefit from denser supervision, and its outcome depends on the privileged information given to the teacher. With reference answers the overall score is 0.277, close to the untrained model's 0.284; with turn-level suggestions it falls to 0.044. Both variants share the distillation loss, frozen teacher, student model and hyperparameters, while turn-level suggestions produce roughly 15-fold larger mean forward KL and raise the mean gradient norm from 0.084 to 1.177.
Perspective
The findings cover four movie-trailer editing task types, with Qwen3.5-9B trained for 50 updates and evaluated on 81 validation tasks from six held-out films; the process-reward conclusions are limited to GRPO+PRM with LLM-judged turn-level rewards.
Whether the process-reward findings extend to untested methods such as GiGPO and Turn-PPO remains open; for on-policy self-distillation the work identifies failure patterns without establishing effective remedies, and improving teacher guidance and training stability requires further study.