# Inference-Time Policy Refinement

Small execution errors in imitation policies accumulate into failure, and earlier fixes relied on additional on-policy learning or costly online planning. A2WM learns local action sensitivity offline from action-perturbed rollouts of a frozen policy, predicting task-relevant geometry as a nominal trajectory plus a linear response to corrections, which turns online refinement into one model evaluation and one convex quadratic program solve. Across four simulated manipulation tasks it reaches the highest mean success among the evaluated baselines, 59.51% versus 27.18% for Diffusion Policy, with 3.8 ms mean refinement overhead per control step excluding base-policy sampling.

Small execution errors in imitation policies accumulate into failure, and earlier fixes relied on additional on-policy learning or costly online planning. A2WM learns local action sensitivity offline from action-perturbed rollouts of a frozen policy, predicting task-relevant geometry as a nominal trajectory plus a linear response to corrections, which turns online refinement into one model evaluation and one convex quadratic program solve. Across four simulated manipulation tasks it reaches the highest mean success among the evaluated baselines, 59.51% versus 27.18% for Diffusion Policy, with 3.8 ms mean refinement overhead per control step excluding base-policy sampling.