Ai2 releases ToyBlocks, a controlled block-and-arm world where simulation-trained robots hit 50.0% zero-shot success

Robot learning gains a deliberately narrow controlled testbed: ToyBlocks limits its subject to a robot arm and blocks on a tabletop, so researchers can change training data or model design, repeat a test and trace why a failure happened. Robot demonstrations are often narrow and do not explain what a robot learned or why it fails, while open-ended domains require modeling so many objects and interactions that realistic physics, automatic generation at scale, reasoning combined with dexterity, and broad in-domain task coverage cannot all be met. ToyBlocks trades breadth for satisfying all four requirements and supplies reproducible evaluation setups: training with objects spawning only in the left region and testing on the right, or training on stack heights of 2 and 5 blocks and testing on heights 3 and 4.

The framework connects data generation, training and evaluation in one simulated setting, with skills produced by reinforcement learning and motion planning and long-horizon tasks assembled by a task-and-motion planner that selects and sequences skills. Such skills would require teleoperation, a person remotely controlling the robot, to collect in the real world; ToyBlocks instead draws on privileged information the simulator supplies directly, such as a block's exact position and orientation, rather than estimating it from camera images. Hundreds of plans are generated and executed in parallel on a single GPU, reaching up to 24 times real-time performance; the initial data release holds 85,336 long-horizon demonstrations, nearly 1,600 hours of robot execution data and 144 million frames.

Evaluation goes beyond success rates to record execution speeds, forces exerted on objects and task progress, measured with the same specifications the planning module uses. Reporting only whether a task was completed makes it hard to separate high-level planning failure from low-level execution failure; ToyBlocks provides reference components at both levels so a failure can be attributed to one of them. In three-letter block-spelling tasks such as stacking "Ai2", Molmo 2 acts as the high-level planner and ManiFlow as the low-level policy, evaluated across three difficulty settings and two generalization conditions.

Models trained entirely on ToyBlocks simulation data were deployed directly on a physical robot, and low-level skill success was close to simulation. Sim-to-real transfer usually needs real-world samples; here both ManiFlow and Molmo 2 saw only ToyBlocks data. Real-world experiments used a Franka FR3 arm with a Robotiq gripper across 30 scene configurations, each with 4 to 10 blocks seen during training; ManiFlow atomic skills scored 80.0% in simulation and 81.1% in the real world, while the Molmo 2 plus ManiFlow hierarchy succeeded in 111 of 200 simulated episodes (55.5%) and 15 of 30 real-world episodes (50.0%).

The demonstrated scope is a simulated tabletop arm-and-block environment and real-world evaluation on a Franka FR3 with a Robotiq gripper after training in that environment; the real-world hierarchy was evaluated on the easiest simulated scene configuration, with block characters visible and upright. The source states that a second release will add a larger-scale dataset, more skills and more comprehensive benchmarking, plus fully integrated reinforcement learning training not contained in the current package.

The source says further trials are needed to establish statistical confidence across tasks for the real-world hierarchy; action-model spatial generalization was poor in the initial results, and the authors believe the problem remains far from solved; the agent solved or correctly declared infeasible 63% of search-and-spell tasks, with errors mostly from confusing look-alike letters (13%) and choosing an incorrect letter orientation (15%).