# Online Reinforcement Learning

Online reinforcement-learning adaptation of long-chunk robot policies previously optimized high-dimensional action sequences directly, which demanded many samples and made real-robot adaptation costly. ExRL instead adapts a scalar decision, how many steps of each predicted action chunk to execute, training a lightweight adapter with off-policy reinforcement learning while the base policy stays frozen. Across simulation benchmarks and four real manipulation tasks it improves diffusion policies, ACT, several vision-language-action models and a world action model with limited online interaction, approaching perfect success on real cube stacking after about ten minutes of physical interaction.

Online reinforcement-learning adaptation of long-chunk robot policies previously optimized high-dimensional action sequences directly, which demanded many samples and made real-robot adaptation costly. ExRL instead adapts a scalar decision, how many steps of each predicted action chunk to execute, training a lightweight adapter with off-policy reinforcement learning while the base policy stays frozen. Across simulation benchmarks and four real manipulation tasks it improves diffusion policies, ACT, several vision-language-action models and a world action model with limited online interaction, approaching perfect success on real cube stacking after about ten minutes of physical interaction.