Long-horizon embodied agents get a stress test: even the strongest models miss clean episodes
Synopsis
MAGMA-Bench evaluates long-horizon embodied agents in dynamic multi-robot environments, adding controlled online perturbations such as execution failures, task interruptions and mission updates, and using a skill-level interface to separate task-level reasoning from perception and low-level control. Earlier benchmarks studied dynamic execution or multi-robot coordination largely one at a time. Even the strongest evaluated models fail on a substantial fraction of clean episodes, success drops further under perturbation, and supervised adaptation greatly improves a small model yet stays substantially lower on longer and multi-robot missions.
Interpretation
Evaluation of long-horizon embodied agents now places dynamic task execution and multi-robot coordination in one environment, with task horizons increasing across episodes. Existing benchmarks investigate dynamic task execution and embodied multi-agent coordination, but provide limited evaluation of their intersection. Evaluation spans increasing task horizons and controlled online perturbations, including execution failures, task interruptions and mission updates, in scenarios requiring coordination across robots with heterogeneous capabilities.
The evaluation separates task-level reasoning from perception and low-level control, so planners of different architectures can be compared under one interface. That skill-level interface is architecture-agnostic, so what is measured is task-level decision making rather than a particular perception or control implementation. Evaluated agents include locally deployable open-weight planners, multi-agent state-tracking strategies and frontier models.
Supervised adaptation greatly improves a small model, but the improved performance remains substantially lower on longer and multi-robot missions. Even the strongest evaluated models fail on a substantial fraction of clean episodes, and their success rates decrease further under online perturbations. Released with the benchmark are 261,646 task-level training examples and an architecture-agnostic expert-trace generation pipeline.
Perspective
The evaluation targets persistent task reasoning in dynamic multi-robot environments, with increasing task horizons and controlled online perturbations including execution failures, task interruptions and mission updates, in scenarios requiring coordination across robots with heterogeneous capabilities. Evaluated agents are locally deployable open-weight planners, multi-agent state-tracking strategies and frontier models. Released with the benchmark are 261,646 task-level training examples and an architecture-agnostic expert-trace generation pipeline.
The summary gives no concrete success-rate numbers for individual models and does not specify the perturbation strengths or the exact task-horizon levels. Supervised adaptation greatly improves a small model, yet performance remains substantially lower on longer and multi-robot missions, a gap the summary does not quantify.