Soft-arm reinforcement learning drops from three years to three hours: a six-dimensional reduced model trains a tracker with 40% lower error than an RNN policy

A trajectory-tracking policy for a soft arm can now be trained solely inside a low-dimensional dynamics model learned from hardware data, with no contact with the arm during training. Model-free reinforcement learning for soft robots previously required extensive hardware exploration, and repeated use of soft material produces memory effects and plasticity that tend to cause hardware failure; soft simulators that capture the nonlinear dynamics are too slow to sit inside the policy-training loop. On the pneumatically actuated SoPrA soft continuum arm, policy training took 3 hours; reproducing the same training through hardware interaction extrapolates to about 3 years.

The training environment is a six-dimensional reduced model described by a polynomial ordinary differential equation, fitted only from pre-collected step-control data. The model restricts the arm's dominant dynamics to an adiabatic spectral submanifold, a low-dimensional attracting invariant manifold whose existence relies on the arm's intrinsic vibration decaying much faster than the control actions change. The model has 6,924 parameters and takes 5 minutes to identify, against 13 hours for the 4.8-million-parameter RNN and 100 hours for the 1.7-million-parameter transformer.

In closed-loop tracking of two-dimensional targets on the hardware, the aSSM-RL policy has 40% lower tracking error than otherwise identical policies trained on the RNN or the transformer. RNN-RL and transformer-RL trace out a track, but as the error grows they become unstable and oscillate, and the arm assumes extreme bending shapes that prevent continuous tracking. The test targets are a lemniscate and a square in the plane, with the square run at three speeds, the fastest 3.2 times the slowest; aSSM-RL has the lowest error on every track, with aSSM-MPC close behind.

The demonstrated setting is the SoPrA, a vertically hanging pneumatically actuated soft continuum arm built from two modules, observed only through camera-tracked tip and mid positions, with closed-loop tasks limited to tracking a lemniscate and a square in the plane; the authors propose next applying aSSM-RL to dynamic dexterous manipulation with soft hands, and a longer-term goal of learning aSSM models from video alone.

The validity of the aSSM model depends on the time-scale separation assumption, which the authors quantify with a non-dimensional slowness measure; theory guarantees it below a threshold, while the measured values are small for the lemniscate and slow square and rise with speed for the faster squares, and closed-loop performance degrades slightly as target speed increases. Commanded and measured chamber pressures show lag and mismatch, producing small wiggles in every policy's closed-loop prediction; a 0.3-second moving average numerically mimics the mismatch-free case, and the authors expect upgraded pressure valves to improve accuracy, but that is a numerical evaluation not yet verified on upgraded hardware.