Space mission planning gets a shared exam: five agent systems on 35 held-out cases, with geometry and constellation design still lagging
Synopsis
AstroAgentBench turns space mission planning into an executable, externally checked agent evaluation: 116 cases across seven task families, with the main evaluation using five held-out cases per family, 35 in total, where an agent must submit a schedule, observation plan or constellation design that an independent verifier checks for schema, timing, geometry, resources and mission value. Among five agent systems, Codex CLI + GPT-5.4 and Kimi CLI + Kimi K2.6 produced verifier-valid plans on all 35 cases, and the strongest systems approach or exceed solver-reference scores on several families while still trailing clearly on geometry-heavy, product-level and design-heavy tasks such as Regional Coverage and Stereo Imaging.
Interpretation
Space mission planning now has a shared executable evaluation: an agent no longer only describes a strategy but must hand over a schedule, observation plan or constellation design that an external verifier judges feasible and scores for mission value. Earlier LLM-for-Space work covered formation planning, satellite scheduling, operations support and on-orbit autonomy, but each used different task contracts, control settings, simulators and success criteria, which made cross-comparison difficult. The benchmark spans seven task families across scheduling, observation planning, constellation design and relay support, with 116 generated or normalized cases; the main evaluation uses five held-out cases per family, 35 in total, and runs five agent systems under the same two-hour budget and the same scientific, astrodynamics and operations-research libraries.
The evaluation separates whether an agent can hand over a feasible plan from how much mission value that plan recovers, so passing a schema check is not the same as completing the mission. Validity alone is too weak: a plan can satisfy the schema and still observe no valuable targets, serve no demand, or cover little area. The verifier first checks submission, schema, time, geometry and resources, then computes family-native value and maps it to a common higher-is-better score; invalid or missing submissions receive zero. On one AEOSSP test case, a valid schedule from Claude Code + Claude Opus 4.6 completes 1,415 of 1,947 tasks and recovers 68.54% of total task weight, yielding a score of 70.87 after turnaround time and energy are folded in; that system's five-case mean is 58.45.
The strong-weak gap falls along specific task families: the strongest agent row scores above the corresponding solver-reference row on SPOT-5, Relay Constellation, Revisit Constellation and SatNet, slightly below it on AEOSSP, and furthest behind on Regional Coverage and Stereo Imaging. The solver-reference rows are quality anchors built by selecting the best and, where available, second-best reference per case and averaging within each family, not certified optima. The strongest agent row scores above the corresponding reference row by 1.00, 3.12, 5.52 and 7.86 points on SPOT-5, Relay Constellation, Revisit Constellation and SatNet, and 3.04 points below it on AEOSSP; the best solver baselines average 86.44 and 96.05 on Regional Coverage and Stereo Imaging, where the strongest agent rows average 73.15 and 68.91.
High-scoring runs build a case-specific solver during the run, calibrate it against verifier feedback, and preserve a verified incumbent. Failures split into two layers: not integrating the physical model, coupled constraints, scored objective and required artifact into the task formulation, and, once formulated, weak search, brittle code, late verifier use or silent goal drift during optimization. Across 175 main-experiment runs, 12.6% end with an invalid submission, 9.1% submit a valid artifact that scores zero, and 22.3% commit to a durable artifact before reading the root workspace contract; workspace support helps conditionally, with procedure injection and memory accumulation paying off only when they supply the missing formulation, calibration or search support.
Perspective
The evaluated unit is the complete agent system, meaning a configured model, its harness and its workspace together; systems sharing a model but differing in harness, tool access, memory or feedback channel count as distinct systems. The main evaluation uses five held-out cases per family, 35 in total, across five systems run under the same two-hour wall-clock budget with 8 cores and 32 GB of memory, and the submitted artifact is scored by an external verifier after the run with no manual repair.
The authors name scale as the main empirical limitation: five cases per family, 35 in total, across five systems, and broader coverage over more cases, systems, seeds and difficulty settings would require many more long interactive runs, with LLM API cost the primary constraint. Solver-reference rows are quality anchors rather than certified optima, so when an agent approaches or outscores a solver row the comparison is to the implemented reference set under the normalized scorer, not to a universal optimum. The ablations cover two forms of workspace support and do not span all harness designs, feedback channels, prompt structures, search tools or memory policies. The benchmark reports one sampled difficulty regime and no scaling curves across larger constellations, denser demand or longer horizons.