Beyond imitation, robots keep learning: RoboFFT reuses noising loss for RL post-training and hits 20/20 on real-robot eggplant grasping

After pretraining on demonstrations, a generative robot policy can keep updating from task success or failure feedback without needing exact action probabilities. Demonstrations do not cover execution-time shifts such as displaced objects, inaccurate grasp angles or a step left unfinished, so the robot meets states rare in training data; PPO-style updates need the probability ratio of the same action under old and new policies, yet diffusion and flow policies produce actions by sampling, and the exact probability of the final action is usually hard to compute directly. On Robomimic Lift, Can, Square and Transport, covering state and image inputs and both flow and diffusion policies, gains were more stable on the harder long-horizon tasks, especially with image inputs.

The update signal comes from the forward noising process already used in pretraining: executed actions are noised and matching losses under the old and new models are compared. Existing methods build optimization signals along the multi-step denoising process, which adds compute and memory cost; RoboFFT instead forms an approximate policy probability ratio from the difference in forward-noising matching loss, then combines it with an advantage estimate in a PPO-style clipped update. The result is a surrogate ratio for optimization, not an exact action probability ratio, so noise is sampled several times and a temperature parameter scales the ratio; for flow policies that usually sample deterministically, controllable noise is added while collecting interaction data so the robot tries different actions.

Training cost is markedly lower than comparable post-training methods, and policy changes concentrate on the decision steps that determine task completion. In training-time comparisons, the flow-policy version of RoboFFT averaged 1.53 times faster than ReinFlow, and the diffusion-policy version averaged 38.0 times faster than DPPO on four state-input tasks; these times cover policy and value-network training and exclude environment data collection. In a diagram analysis of one successful Square execution, at identical trajectory states the post-trained policy showed higher action-quality metrics and lower sampled-action diversity at key stages such as grasping, carrying and alignment, with action quality coming from an auxiliary value model rather than per-action real-robot success rates.

The real-robot pipeline runs offline to online: a flow policy is pretrained on teleoperated demonstrations, then trained offline with reinforcement and imitation learning on robot-collected trajectories, and finally updated through online interaction, with success or failure judged by a human. Real robot interaction is slow and data collection costly, so online trial and error alone is impractical; at the same data budget, merely pooling data and continuing behavior cloning reached 14/20, indicating the gain cannot be explained by more data alone. Peg insertion, eggplant grasping, pouring and cabinet retrieval were each tested 20 times; on eggplant grasping, RoboFFT reached 18/20 after offline post-training and 20/20 after online post-training.

Validation covers Robomimic Lift, Can, Square and Transport simulation tasks, post-training of the flow action head of vision-language-action model π₀ on LIBERO-Object and LIBERO-Spatial, and four real-robot tasks (peg insertion, eggplant grasping, pouring and cabinet retrieval) with 20 trials each; real-robot task feedback is judged success or failure by a human.

What is obtained is a surrogate ratio for optimization, not an exact action probability ratio; action-quality metrics come from an auxiliary value model, not per-action real-robot success rates; too little exploration noise makes effective exploration hard and too much also hurts success, with a medium level balancing better on that task; raising the sample count from 1 to 2 helped converge faster while further increases gave limited benefit, an observation under that task and policy setting that does not imply all generative policies need only few samples.