Adapting long-chunk robot policies by learning how long to execute: ExRL keeps base policies frozen and nears perfect success on real cube stacking after about ten minutes
Synopsis
Online reinforcement-learning adaptation of long-chunk robot policies previously optimized high-dimensional action sequences directly, which demanded many samples and made real-robot adaptation costly. ExRL instead adapts a scalar decision, how many steps of each predicted action chunk to execute, training a lightweight adapter with off-policy reinforcement learning while the base policy stays frozen. Across simulation benchmarks and four real manipulation tasks it improves diffusion policies, ACT, several vision-language-action models and a world action model with limited online interaction, approaching perfect success on real cube stacking after about ten minutes of physical interaction.
Interpretation
Online adaptation of long-chunk policies no longer optimizes the whole high-dimensional action sequence; it learns how long to execute each predicted chunk before replanning. Adapting such policies with reinforcement learning previously required many online samples because the optimized action chunks are high-dimensional, while real-robot interaction data are slow and costly to collect. On the Robomimic, Adroit and LIBERO simulation benchmarks, ExRL often raises success rates of frozen base policies to near perfect, and clearly exceeds the best fixed execution length on DP Can, Square and Tool Hang.
The adapter outputs only a discrete execution length, while the base policy still produces the primitive actions, so the adaptation space does not grow with chunk length. Named ExRL, the approach adds a lightweight execution-length policy on top of the base policy, taking the current state and the predicted chunk as input and choosing an execution length from the feasible set. The execution-length policy is trained with off-policy algorithms, instantiated as the value-based ExRL-Q and the discrete-SAC ExRL-SAC, using a temporal-difference target that accumulates reward and discount over the actual execution duration.
On real robots, ExRL-Q with a frozen base policy approaches perfect success on cube stacking after about ten minutes of physical interaction, and nearly completes long-horizon clothing folding within minutes. The four real tasks cover contact-rich power insertion, deformable-object folding, long-horizon coordination and multi-stage parts organization, and final evaluations show substantial gains over fixed-length execution on all four. Real experiments use an AgileX Piper arm; ExRL-Q selects execution length from the mean of ten critics, and the operator only marks task or stage completion without action corrections or teleoperation.
Perspective
The demonstrated scope is several action-chunked policy classes on three simulation benchmarks plus four real tasks on an AgileX Piper arm; the real experiments keep the base policy frozen and do not include online behavior cloning.
Several real tasks still fall short of success, and performance remains constrained by the actions the base policy can propose; each task is adapted separately, with joint multi-task adaptation and generalization to unseen tasks not yet evaluated; ExRL has also not been tested together with asynchronous inference methods such as real-time chunking.