Frozen generative policies need not be rebuilt: CANDO corrects in both noise and action space, reaching 91.4% final success
Synopsis
Pretrained generative policies give robot control strong behavior priors, but adapting them efficiently with online reinforcement learning remains hard: noise-space residual methods regulate adaptation through fixed latent feasible regions that may exclude valuable noise edits, while action-space residual methods are more sample-efficient yet still leave room for improvement. CNDO drops predefined latent feasible regions, learning state- and noise-conditioned directions regulated by a soft conservative constraint built from conservative value estimation and gated execution; CANDO adds an action-space residual on top. Experiments report 76.1 NAUC and a 91.4% final success rate for CANDO, 6.6 NAUC and 4.1 percentage points above the best single-space results.
Interpretation
When adapting a frozen generative policy, there is no need to predefine a latent feasible region to limit noise-space changes; instead the policy learns which direction to move in noise space. Earlier noise-space residual methods typically constrained adaptation with fixed latent feasible regions, which may shut out valuable noise edits. After removing predefined latent feasible regions, CNDO narrows the gap between noise-space and action-space residual learning, providing a basis for later two-space combination.
Concretely, it learns state- and noise-conditioned directions that shift the frozen policy's input noise around Gaussian latent samples, with a soft conservative constraint controlling the size of the change. The soft conservative constraint consists of conservative value estimation and gated execution, replacing the earlier hard feasible-region boundary. CANDO, which adds an action-space residual on top of CNDO, combines structured latent-space improvement with local action correction and outperforms either single-space optimization in the experiments.
The payoff of the two-space combination shows in the metrics: CANDO reaches 76.1 NAUC and a 91.4% final success rate. That is 6.6 NAUC and 4.1 percentage points of success rate above the best single-space results. The comparison is against the best single-space result under the same experimental setting, not an arbitrary baseline.
Perspective
The demonstrated scope is residual reinforcement learning experiments on frozen generative policies, evaluated by NAUC and final success rate; the source is summary-level and gives no detail on the task set, environments or training sample size.
The source is presented as a summary, and the specific tasks, environments and training scale behind the 76.1 NAUC and 91.4% final success rate are not described in the visible text; whether the two-space gain over single-space holds across different settings is not stated.