Frozen world models skip retraining: Library of Hands calibrates only the "hand" and lifts success from 28.6% to 62.3% when hands change and return
Synopsis
When a camera moves or a controller is replaced, the same command moves the robot in a different direction or by a different magnitude, while a frozen world model still plans under its training-time action convention, so every plan is expressed in the wrong action frame. Library of Hands (LoH) trains nothing and leaves the world model frozen: it uses the frozen predictor to infer the applied action from two successive camera images, estimates the map from commands to applied actions, and stores each identified map in a library for reuse. In closed-loop tests where hands change and come back, LoH raises a frozen planner's success from 28.6% to 62.3%, and loses only 0.6 percentage points when nothing changes.
Interpretation
LoH lets a frozen world model keep planning after the action convention changes, by correcting only the map from commands to applied actions rather than the model itself. With the LeWM world model, a frozen planner succeeds in 77.5% of episodes under its original convention and in only 7.8% when its commands are turned by 90 degrees. In closed-loop tests where hands change and come back, LoH raises overall success from 28.6% to 62.3%; on the 12 seeds that also ran the true-hand arm, LoH closes 69.2% of the gap to the true hand.
LoH treats the frozen predictor as a sensor: it finds the action that best explains the change between two successive camera images, then compares that reading with the command to read the current hand. Each reading is weighted by the uncertainty implied by the Jacobian of the predicted state with respect to the action, so a flat Jacobian counts for little; exact Bayesian evidence then selects among 30 models formed by five map classes (identity, turn-and-scale, mirror, linear, affine) and six delays. On the three simulated tasks PushT, Reacher and Two-Room, the same method code runs unchanged with three world models, LeWM, DINO-WM and JEPA-WM; in the RoboCasa simulated kitchen, two of three models pretrained on real robot data pass the pre-set criteria.
Perspective
The demonstrated scope is closed-loop simulation: LeWM, DINO-WM and JEPA-WM on the PushT, Reacher and Two-Room tasks, plus three models pretrained on real robot data in the RoboCasa simulated kitchen; the hand changes are injected, and no real robot has been tested.
When the hand switches inside an episode, LoH has a mean gain of 6.6 percentage points, below its pre-set bar of 10 percentage points; when the world changes and the hand does not, LoH loses 5.0 percentage points and misses its bar. In RoboCasa, LoH gains only 7.8 percentage points on JEPA-WM and costs 6.2 percentage points with no change, so two of the three models pretrained on real robot data pass the criteria.