A robot policy misread "in the bowl": closed-form weight edits, with no retraining or instruction change, lift held-out success from 27/128 to 106/128
Synopsis
A zero-shot vision-language-action policy told to put the cube then the banana in the bowl releases the cube early because it binds "in the bowl" to the banana rather than the cube in its gripper. Closed-form instruction distillation writes the per-layer effect of a working rewording directly into the VLM's MLP weights, with no gradients, training data or change to the instruction; on 128 held-out seeds success rises from 27/128 to 106/128, while a random-direction update of the same size leaves success unchanged.
Interpretation
A generalist robot policy never trained on this task can be repaired with its instruction left exactly as written: it previously let go of the cube in transit and now keeps holding until the cube reaches the bowl. The usual remedy is to collect demonstrations and fine-tune; weight-space programming offers a gradient-free alternative, but existing robot results came mostly from checkpoints fine-tuned per task, leaving open which failures of a zero-shot policy such edits can fix. On 128 held-out seeds unused during editing, success rises from 27/128 to 106/128, with 81 starts fixed and 2 broken; a random-direction update with the same layers, rank and singular values scored 9/64 on the first 64 of those seeds, in line with the unedited policy's 13/64 and 9/64 in two runs.
The repair first finds a rewording under which the policy succeeds, then writes that rewording's internal effect into the vision-language model's weights. Closed-form instruction distillation forms 72 keys per layer, each the mean input to that layer's MLP down-projection under the original instruction over 6 recorded premature-release decisions, and values equal to the explicit rewording's per-layer residual increment minus the original's, combined by a regularized minimum-norm update that maps keys to values. Offline replay shows the edit removes all 21 premature releases while leaving all 9 recorded correct carries and all 19 phase-two decisions unchanged; on 32 practice seeds closed-loop success rises from 3/32 to 25/32, after which the edit was frozen and scored on held-out seeds.
Perspective
The conclusions rest on one zero-shot checkpoint in simulation: the DROID version of the policy evaluated on 16 RoboLab tasks, with 16 to 128 pairs per comparison and the instruction text held fixed. The released weightsmith distill tool has so far run end to end on one task only and needs the known-good rewording as input.
The authors note that the full edit is low-rank but not sparse, changing 13.4% of the policy's stored values, and the 0.16% variant still changes 7.4% once applied; the edit-location comparison covers one task and one wording; transfer was not predicted in advance, so deployment would need broad regression checks; and the failure levels were assigned by editor diagnoses without an independent rater. Reruns with unchanged weights also flip about a third of paired outcomes, so evaluations should pair seeds, rerun the unedited policy and confirm gains on held-out seeds.