ACT3 tri-stream transformer makes the action expert the sole integrator of semantics and dynamics, lifting RoboCasa success from 59.4% to 71.2%
The authors introduce ACT3, an action-centric tri-stream transformer in which a semantic stream (VLM) and a dynamics stream (world model) compute independently while only the action expert reads both through layerwise attention, with all streams jointly optimized under action supervision. It reaches 71.2% success on 24 RoboCasa kitchen tasks versus 59.4% for the baseline, 98.6% average on LIBERO versus 96.9%, and 82.4% versus 68.4% on LIBERO-Plus distribution shifts.