Bimanual robots get stereo data without stereo cameras: warp the left view, inpaint the holes, lift transparent-bottle grasping from 11/20 to 18/20

Stereo training data for bimanual robots can be synthesized from existing monocular demonstrations instead of captured with stereo cameras. Large-scale robot datasets are overwhelmingly monocular and single-arm, and the few stereo-camera datasets are single-arm, so no large-scale bimanual stereo dataset exists; collecting one at comparable scale would require stereo hardware and re-teleoperation for each new task. On an AgileX PiPER bimanual platform, mid-training ExStereo on StereoEngine-L data succeeds on 50/60 trials across three tasks, versus 36/60 with no mid-training, 40/60 for StereoCrafter and 31/60 for SVG; the largest gain is transparent-bottle lifting at 18/20 versus 11/20 without mid-training.

The right view is synthesized by shifting the left view horizontally according to depth, then letting a video diffusion model fill only the holes that shift leaves behind. Existing monocular-to-stereo methods mostly target VR or driving content and typically need stereo pairs for training, while reconstruction-based novel-view synthesis needs multi-view data and per-scene optimization and does not fit a dynamic single-view setting. A monocular video depth model produces inverse depth, forward splatting moves pixels to the right camera position at a chosen median disparity, and uncovered pixels become a disocclusion mask; the inpainter is fine-tuned only on 3,469 monocular robot episodes with no stereo pairs, using masks produced by the same warp plus partial arm masks and disparity dropout.

When no real video exists, both photorealistic views can be generated jointly from a simulated trajectory without any real stereo correspondence. Prior robot video generation produces only monocular video, and simulation rendering differs substantially from the real world in texture, materials and lighting. A second virtual camera in RoboTwin renders a true stereo pair, each frame is converted to Canny edges to strip the simulator appearance, and the tiled left and right frames are generated zero-shot by Wan2.2-Fun-A14B-Control; on Stack Two Bowls, Lift Two Bottles and Container Plate, fine-tuning ExStereo on this data reaches 11/20, 13/20 and 15/20, while training on simulated ground-truth point clouds gives 0/20.

The method targets training-data generation for bimanual manipulation policies and is validated in the real world on an AgileX PiPER bimanual platform with a ZED Mini 2 camera, across five tasks: Stack Two Bowls, Lift Transparent Bottles, Close Door, Lift Two Bottles and Container Plate. StereoEngine-L fits settings where monocular demonstrations already exist for the target embodiment, while StereoEngine-S fits settings with no real video where the task can be built in simulation. Generation quality is assessed on a self-built set of real stereo captures, since no stereo benchmark exists for robotics.

Video diffusion models still struggle with transparent and highly reflective objects, which can degrade the generated stereo pairs; StereoEngine-L inherits the quality of the monocular depth estimator, so inaccurate depth yields geometrically inconsistent right views, and its inpainting quality depends on the amount of fine-tuning data; StereoEngine-S relies on Canny edges from simulation, so larger camera distances produce noisier edges that can degrade the generated views. Policies trained on simulated ground-truth point clouds reach a 0% success rate on real tasks, and perturbing each point with isotropic Gaussian noise did not yield meaningful improvement.