Ai2 releases ToyBlocks, a controlled robot-learning sandbox: skills trained in simulation hit 81.1% on real hardware versus 80.0% in sim
Synopsis
Ai2 released ToyBlocks, an open simulation framework built around a robot arm and tabletop blocks, the first controlled world in its FoundationWorlds program. It automatically generates tasks and demonstrations, letting researchers vary training data, repeat experiments and trace why a robot fails. In physical tests, the low-level skill model ManiFlow, trained only on simulation data, reached 80.0% success in simulation and 81.1% on the real robot, while the combined high-level planner plus low-level executor succeeded in 55.5% of simulated episodes and 50.0% of real-world episodes.
Interpretation
Robot learning gains a deliberately narrow, controlled simulated world where researchers can change training data, repeat tests and trace the cause of failures instead of relying on narrow real-robot demonstrations. Robot demonstrations have often been narrow and do not explain what a robot learned or why it fails, and open-ended domains make it infeasible to satisfy realistic physics, large-scale automatic task generation, reasoning combined with dexterity, and broad task coverage at once. ToyBlocks' narrow scope makes it possible to meet all four requirements while keeping the simulated arm and blocks behaving very similarly to their physical counterparts.
The framework uses reinforcement learning and motion planning to generate short-horizon skills such as pushing and flipping, then a task-and-motion planner selects and sequences them to produce labeled demonstration data in bulk. Such skills would normally require time-consuming teleoperation to collect in the real world and are hard to scale. Skill training runs in Nvidia's Isaac Lab and can draw on privileged information the simulator supplies directly, such as a block's exact position and orientation; hundreds of plans execute in parallel on a single GPU at up to 24 times real-time performance.
Physical validation shows models trained entirely on ToyBlocks simulation data can be deployed zero-shot on a Franka FR3 arm. Sim-to-real transfer usually needs fine-tuning on real data, whereas both models here were trained only on simulation data. ManiFlow's atomic-skill success was 80.0% in simulation and 81.1% in the real world; the Molmo 2 plus ManiFlow hierarchical system completed 55.5% of simulated episodes (111/200) and 50.0% of real-world episodes (15/30), with real-world tests spanning thirty scene configurations.
Perspective
The demonstrated scope is a tabletop arm and blocks performing pushing, flipping, rotating, pick-and-place and three-letter spelling tasks; real-world experiments use a Franka FR3 arm with a Robotiq gripper across thirty scene configurations, each containing 4 to 10 blocks seen during training. The intended readers are robot-learning researchers who want to vary training data or model design and trace failure causes. The source says a second release is targeted in a few weeks with a larger-scale dataset, more skills and more comprehensive benchmarking, plus fully integrated RL training not contained in the current package.
The source states that the hierarchical system's real-world evaluation covers only 30 episodes and that further trials are needed to establish statistical confidence across tasks. It also says action-model generalization remains far from solved and that prior studies on it report conflicting evidence.