Why Do Robots Fail at Stacking Blocks? A Virtual Testbed Called ToyBlocks Aims to Find Out

ToyBlocks, an open virtual testbed for studying how robots push, flip and stack blocks, was released on November 6. In a post on the Hugging Face blog, the researchers said the environment lets you vary training and evaluation conditions using a robot arm and blocks on a simulated tabletop. It does not simply check whether a task succeeded. The point is to look at what experience the robot learned from, and where exactly it failed.

Why test with blocks?

ToyBlocks is the first instance of FoundationWorlds, a concept that bundles task generation, training demonstrations and behavior measurement tools into a controlled virtual environment. The researchers explained that narrowing the domain makes it possible to render motion and appearance close to reality while still running many tasks repeatedly. They plan to extend the approach to more complex environments in the future.

Even handling blocks requires several judgments. The robot has to find and sort the pictures and letters on the block faces, or arrange them in a given order. To check a letter it cannot see, it has to turn the block first. Block size and table height can also be changed, to test whether a familiar motion still works under new conditions.

How do you collect training examples without people?

A feature that automatically generates training demonstrations reduces the burden of human operation. Instead of having people teleoperate a robot to collect examples, the system records the process of completing tasks in simulation. For learning short motions, it uses Isaac Lab, Nvidia’s robot learning toolkit. The researchers said that with a single graphics processing unit (GPU) they can generate and execute hundreds of plans in parallel, collecting data up to 24 times faster than real time.

Image: huggingface.co

Researchers can also control what experience a robot learns from, and how much of it. One approach is to increase the number of flipping examples, or mix in rotation and pick-and-place examples, then compare the effects on tasks that were not used in training. Evaluation does not stop at success rate. It also measures execution speed, the force applied to objects, and how far the task progressed. The goal is to see how behavior shifts depending on training conditions.

Does it work in positions it was never trained on?

In early experiments, even heavily trained models proved weak under unfamiliar spatial conditions. The researchers tested ManiFlow, π0.5 and MolmoAct 2, separating new layouts inside the training distribution from layouts outside it. For example, training on the left side of the table and evaluating on the right, or training on stacks of 2 and 5 blocks and then asking for stacks of 3 and 4. The researchers said these models generalized poorly to new conditions. They also analyzed how well various training techniques intended to improve this actually worked.

Image: huggingface.co

For multi-step tasks, planning and motion are evaluated separately. Spelling “Ai2” with blocks requires finding hidden letters and deciding the stacking order. In the experiment, Molmo 2 chose the blocks and actions while ManiFlow moved the robot arm. By comparing against reference components that substitute in the correct plan or the correct motion, it becomes possible to tell whether a failure came from judgment or from execution.

Even without robotics expertise, you can connect a model such as Claude to the web interface and give instructions for tasks in everyday language. The researchers also showcased an experiment in which Claude Opus 5.5 looked at camera footage and called tools to find and arrange blocks. The team additionally ran experiments transferring models trained in the virtual environment onto real robots. They confirmed comparable success rates on physical hardware, but explained that more experiments are needed before they can be confident across the full range of tasks.