# Policy Adaptation

Online reinforcement-learning adaptation of long-chunk robot policies previously optimized high-dimensional action sequences directly, which demanded many samples and made real-robot adaptation costly. ExRL instead adapts a scalar decision, how many steps of each predicted action chunk to execute, training a lightweight adapter with off-policy reinforcement learning while the base policy stays frozen. Across simulation benchmarks and four real manipulation tasks it improves diffusion policies, ACT, several vision-language-action models and a world action model with limited online interaction, approaching perfect success on real cube stacking after about ten minutes of physical interaction.

Online reinforcement-learning adaptation of long-chunk robot policies previously optimized high-dimensional action sequences directly, which demanded many samples and made real-robot adaptation costly. ExRL instead adapts a scalar decision, how many steps of each predicted action chunk to execute, training a lightweight adapter with off-policy reinforcement learning while the base policy stays frozen. Across simulation benchmarks and four real manipulation tasks it improves diffusion policies, ACT, several vision-language-action models and a world action model with limited online interaction, approaching perfect success on real cube stacking after about ten minutes of physical interaction.

VEPA writes an agent guardrail as executable symbolic rules, has an LLM propose patches from deployment feedback while a deterministic verifier admits only structurally valid, conflict-free policies, and reaches the highest F1 on events covered by its active policy across three agent safety benchmarks while expanding the scope of deterministic enforcement during deployment.

VEPA writes an agent guardrail as executable symbolic rules, has an LLM propose patches from deployment feedback while a deterministic verifier admits only structurally valid, conflict-free policies, and reaches the highest F1 on events covered by its active policy across three agent safety benchmarks while expanding the scope of deterministic enforcement during deployment.

The work develops a finite-depth framework that estimates policy sensitivity by approximating the policy Hessian and mixed derivative from reference-environment information, with an adjustable propagation depth determining where derivative propagation along the trajectory is truncated; the authors characterize the omitted derivative contributions and derive truncation-error bounds that are nonincreasing with depth and vanish at full-horizon propagation, and in a belief-driven pursuit-evasion game they find that derivative-estimation errors generally decrease with depth, that the method outperforms baselines in estimation accuracy and policy adaptation, and that sensitivity-based initialization improves zero-shot return over direct transfer and shows advantages for subsequent fine-tuning.

The work develops a finite-depth framework that estimates policy sensitivity by approximating the policy Hessian and mixed derivative from reference-environment information, with an adjustable propagation depth determining where derivative propagation along the trajectory is truncated; the authors characterize the omitted derivative contributions and derive truncation-error bounds that are nonincreasing with depth and vanish at full-horizon propagation, and in a belief-driven pursuit-evasion game they find that derivative-estimation errors generally decrease with depth, that the method outperforms baselines in estimation accuracy and policy adaptation, and that sensitivity-based initialization improves zero-shot return over direct transfer and shows advantages for subsequent fine-tuning.