Dream4ACT unifies multi-embodiment video-action modeling with fixed-view action views, reaching 88.98% average success on RoboTwin 2.0

It introduces "action views," a fixed-shape multiview visual action representation: URDF-based forward kinematics and rendering turn current and target joint configurations into images from four fixed virtual cameras (front, top, left, right) whose tensor shape is independent of joint dimensionality while retaining embodiment-specific joint geometry. Earlier visual interfaces mostly encode end-effector quantities (e.g., SpatialVAM's virtual-view heatmaps, Action Images' multiview images) or encode actions as latent frames or skeleton conditioning; this work explicitly encodes the full arm and gripper configuration, and its virtual cameras are independent of physical observation cameras, so action-view construction needs no physical-camera extrinsic calibration. The method section gives the action-view definition and rendering conventions (base-to-arm links yellow, arm nodes blue, grippers green-to-red for open-to-closed), and Appendix A.2 describes saving one task_rig.json per dataset-embodiment-task group, fixing extrinsics, and selecting intrinsics from an occupancy bound over projected training-frame joints; in the ablation, on the same recorded joint-state sequences, action-view rendering improves PSNR from 23.19 to 24.33 dB, SSIM from 0.899 to 0.906, and LPIPS from 0.105 to 0.093 on DROID relative to camera-aligned skeleton conditioning.

A shared video autoencoder and diffusion transformer jointly model RGB observations and four action-view streams under masked flow matching, with modality/view embeddings and sequence-specific RoPE offsets separating streams, so one set of weights supports forward dynamics, inverse dynamics, and joint generation. Existing unified video-action models use separate diffusion decoders with masked inputs (UVA), couple video and action diffusion with independent noise levels (UWM, Pelican-Unify 1.0), or rely on modality-specific heads or experts; this work replaces the action representation itself with visual streams, letting heterogeneous joint spaces share one tokenization and prediction interface, sampling the three modes at 0.2/0.2/0.6 and I2V versus V2V temporal-prefix conditioning at 0.7/0.3 during training. The paper gives the clean/noised latent construction, the velocity-field loss restricted to corrupted positions, and the table of stream sets per mode; Appendix A.1 lists batch size 16, AdamW, cosine schedule, 1000 warmup steps, simulation action chunk 40, and real-world action chunk 80.

It proposes training-free, URDF-constrained multiview action recovery: predicted action views are binarized and dilated, candidate configurations are scored by IoU plus one-sided Chamfer distance against URDF renderings, and frame-wise LM/Gauss-Newton estimation with cubic B-spline window-level smoothing and color-decoded grippers yields executable joint targets without a learned embodiment-specific action decoder. Where Hydra-0 uses a trained action head, Masked Visual Actions a learned inverse dynamics model, and SpatialVAM learned rotation and gripper decoding, this work delegates recovery to geometric matching; where BridgeV2W and GeniWorld condition on URDF-rendered motion aligned with observation viewpoints, this work uses fixed virtual cameras, removing physical-camera extrinsics from both action-view construction and recovery. The paper gives the scoring function (an IoU term plus a Chamfer term), the recovery procedure table, and Appendix A.5 recovery errors from ground-truth action views on 31 shared tasks with five held-out trajectories per task: Aloha-Agilex 1.39 mm position / 1.57 deg rotation, Piper 1.66 mm / 1.71 deg, ARX-X5 1.22 mm / 1.95 deg, Franka-Panda 4.11 mm / 9.45 deg, and UR5-Xsg 11.85 mm / 21.39 deg.

Closed-loop and generation evaluations show one checkpoint executing across embodiments: 88.98% average success over 50 RoboTwin 2.0 tasks (90.50% clean, 87.46% randomized), a TriWorldBench overall score of 65.66 in forward dynamics mode, and 85.0%-90.0% mean success on the two common real-world tasks place_block and wipe_plate. The authors report that this average exceeds one cited baseline by 7.76 and 10.7 percentage points and is slightly above Motus, while LingBot-VA and Fast-WAM score higher; on TriWorldBench it is comparable to BWM's 65.54 and obtains the highest reported P3D, MQ, and TC among the listed methods, tying BWM on VQ. Simulation follows official RoboTwin 2.0 task-success criteria with 100 trials per task per configuration; the multi-embodiment evaluation uses a checkpoint jointly trained on five embodiments over the 31-task intersection with 25 trials per task per robot (775 trials per setting); real-world data comprise 60 demonstrations per embodiment-task pair, 720 in total across 12 pairs; TriWorldBench reuses the same checkpoint without benchmark-specific fine-tuning and follows the official submission protocol for head and wrist videos.

The interface targets joint-controlled robotic manipulation: training and evaluation cover five simulated embodiments (Aloha-Agilex, ARX-X5, Franka-Panda, Piper, UR5-Xsg) and four real-world platforms (Aloha-Agilex, TienYi2.5 Pro, Franka Research 3, UR5e), with real-world tasks place_block, wipe_plate, and the bimanual stack_blocks and storage_item. It applies when the embodiment's URDF and rendering conventions match training, with virtual-camera presets fixed and reused across frames and episodes and test trajectories excluded from preset determination. Researchers and engineering teams who want a single shared set of weights across kinematic structures and can accept chunk-based execution can reuse this representation directly; its recovery pipeline requires no learned embodiment-specific decoder.

Recovery accuracy depends on the fidelity of generated action views and on geometric observability, and also on consistency between the URDF and training rendering conventions, so kinematic mismatch can affect physical execution; the authors note that Franka-Panda and UR5-Xsg are assembled in simulation from two single-arm robots rather than designed as integrated dual-arm systems, and that this integration may affect inter-arm coordination and joint-target tracking, plausibly contributing to their lower success rates. Real-world evaluation uses each platform's nominal background without background variation or additional distractors, and task coverage is limited. On latency, the real-world setup takes roughly 6.9 s for generation and 2.7 s for recovery per 80-step chunk, so the current implementation supports chunk-based execution rather than real-time per-step replanning; broader task coverage and transfer to unseen embodiments remain to be validated. In addition, some baseline cells in the TriWorldBench comparison table are empty in the supplied material, so comparisons can only be made against the values the text explicitly reports.