Reading plans from actions: GenEA folds body mechanics and object physics into one state space to explain human judgments
Synopsis
GenEA places an agent's coarse body biomechanics and its physical interactions with nearby objects into a multilevel state space, then embeds that space in a generative model to yield a joint probability distribution over goals and embodied plans. Earlier accounts of how people reason about plans mostly abstracted agents as point-like entities in two-dimensional grid worlds, which cannot carry the body and terrain constraints of natural settings. Across block-manipulation and obstacle-course navigation experiments, simple transformations of that joint distribution reproduce human goal inferences, action inferences, one-hand versus two-hand strategy preferences and difficulty judgments, outperforming a task-and-motion-planning baseline, heuristic models and a frontier vision-language model.
Interpretation
GenEA moves the reading of other people's plans from point-like agents in grid worlds to natural settings that include a body, objects and gravity. Earlier computational accounts abstracted the agent into a point-like entity operating in a two-dimensional grid-like environment under highly constrained transition rules, whereas naturalistic embodied settings are open-ended and dynamic. Across two qualitatively distinct domains, block manipulation and obstacle-course navigation, GenEA aligned closely with human judgments and outperformed heuristic models, standard task-and-motion planning and a frontier vision-language model.
The construction adds a dynamics layer on top of the two-level robotics task-and-motion-planning structure, so planning searches in a loop through symbolic, kinematic and physical levels. Standard task-and-motion planning contains only a symbolic and a kinematics level; GenEA adds an intuitive-physics level implemented with game-engine-style approximate simulation and treats trajectory endpoints as noisy rather than fixed. Manipulation uses the PhysX engine and navigation uses MuJoCo, the navigation agent is the Talos humanoid restricted to joints from the hips downward, and plan probability follows a Boltzmann distribution over execution cost.
Human judgments about plans can be read as transformations of this joint distribution without a separate set of inference rules. Goal inference, action inference, one-hand versus two-hand preference and difficulty ratings correspond to different query constraints, and the model reports preferences through one parameter-free choice rule. In three preregistered experiments, GenEA correlated significantly with human preferences and response times, matched the direction of human sensitivity to platform material, while the task-and-motion-planning baseline showed no sensitivity to that material change.
Perspective
The demonstrated scope is block-rearrangement manipulation tasks and navigation tasks of walking and jumping between platforms, with human data from preregistered online behavioral experiments and model inputs limited to the initial and goal object configurations that participants also saw. The authors frame GenEA as a computational-level account rather than specifying the algorithm by which the mind implements that distribution.
The authors state that the mind may approximate GenEA's posterior distribution through a collection of smaller forward models and modules, and their claim is that any such representation must be broadly analogous to GenEA. Using GenEA in robots would also require perception systems that make it image-computable, which is proposed rather than demonstrated.