Robot actions without a learned action module: project the tool frame into the wrist camera and let a pretrained VLM emit executable motion

The physical meaning of a robot action is placed directly into the picture the vision-language model already sees, so the model can emit executable motion without first learning a robot-specific action representation. Prior vision-language-action models either treat actions as extra tokens or attach flow-matching, diffusion, or regression action heads, and the semantics of those action representations must first be taught from robot data pretraining. On a UR3 arm, a 27B-scale model reaches 79.4% average zero-shot success on the simpler task group, while the compared VLA baselines need roughly 30 demonstrations of fine-tuning to match that level.

Actions are generated as plain text, so the model's own autoregressive language decoder serves as the action decoder. Earlier approaches introduce dedicated action tokens or a separate continuous-action predictor, whereas this interface uses only symbols already present in the pretrained vocabulary. An action is written as a text string with explicit signs and two decimal places, for example 0|+0.40,-0.10,+0.91,+6.00|+0.00,+0.00,+1.00,-45.00|+1.00, whose four fields denote task completion, translation, rotation, and gripper state.

The interface is validated on a 6-DoF UR3 with a wrist-mounted RGB-D camera and an external camera over a tabletop workspace, across rigid, articulated, deformable, precise, and long-horizon tasks; zero-shot capability concentrates on the simpler task group, and the harder group is addressed by fine-tuning on robot data. Cross-embodiment transfer is tested between the UR3 and a 7-DoF Flexiv sharing the same gripper, and that experiment does not use external-camera observations. The authors propose next steps including characterizing zero-shot capability and reasoning budgets across more open-source and proprietary VLM families and parameter scales, extending the native textual action interface to bimanual and mobile manipulation, and investigating whether scaling pretraining with that interface improves visual-spatial reasoning and zero-shot manipulation.

Zero-shot evaluation shows non-trivial capability only on the simpler task group, while the harder group generally fails zero-shot and requires fine-tuning. Ablations show that removing either the action-frame projection or the metric depth information drops average success from 79.4% to 0%, making both necessary in the current setting. Grammar-constrained decoding constrains only command syntax, and the numerical action is still predicted by the model; in the LIBERO simulation study no depth information is used, and the model receives the same input observations as the compared VLA baselines. The authors also note that many open questions remain around this new action interface.