Robot rewards stop chasing lighting and camera angle: task-relevant dynamics lift MetaWorld success to 94.3%

In robot manipulation reinforcement learning, dense rewards learned from video can now respond to changes in task execution itself rather than to camera shifts, lighting, or background variation that do not matter for the task. Earlier video rewards were computed in entangled visual spaces where task-relevant state changes were confounded with extraneous visual variation, so the reward could respond to visual distractors instead of actual task execution. Across 16 visual manipulation tasks on MetaWorld-v3, DiTaR reaches 94.3% final success versus 65.2% for the strongest learned-reward baseline ReWiND; on ManiSkill it reaches 87.5% versus 74.6% for Env-only and 55.0% for ReWiND.

The reward comes from how the agreement between observed transitions and the learned task dynamics changes over time in the task-relevant latent space, giving a directional dense signal rather than an absolute score. An absolute compatibility score says whether an observation agrees with the learned distribution, but not whether execution is improving relative to its recent history, which is why the temporal change is used. On MetaWorld and ManiSkill, a variant without temporal differencing learns more slowly on Coffee Pull, Faucet Open, and ManiSkill Push Cube, and removing history conditioning makes little progress on Handle Pull.

The task-relevant representation is learned from multi-frame video latents through an asymmetric two-stage procedure in which language first supplies task semantics and is then removed, forcing a second latent to recover task-relevant information from video. A single-stage reconstruction-based encoder without language conditioning, used in place of the task-relevant latent, drops final success from 94.3% to 67.2% and raises NormT@0.8 from 0.244 to 0.458. On the 12 MetaWorld-v3 tasks, DiTaR with task-relevant latents reaches 94.3% final success and 91.7% SR80, compared with 67.2% and 58.3% for the reconstruction-based variant.

DiTaR targets researchers and engineers learning dense rewards from action-free videos for robotic manipulation RL, validated on 16 visual manipulation tasks from MetaWorld-v3 and ManiSkill and transferred to real-world offline RL through reward relabeling on 150 demonstrations collected with a Franka Research 3 robot. During policy learning it still uses the sparse environment reward, with the DiTaR reward added as a dense shaping signal alongside an RND exploration bonus.

DiTaR still relies on sparse environment rewards during RL, leaving fully video-based reward learning for future work. Its pretraining data are dominated by limited-view robotic manipulation videos, and broader multi-view and egocentric data may further improve generalization across diverse manipulation tasks and environments.