Agents play full Slay the Spire runs: top models match experts in act one, then fall behind
Synopsis
Agentic large language models handle complex tasks, yet long-horizon strategic reasoning lacks a dissectable benchmark. SpireBench has 20 LLMs play the complete commercial deck-builder Slay the Spire at its highest difficulty: frontier models beat heuristic baselines and match the average human player but remain far behind experts, and the gap widens as runs lengthen, with the strongest models winning act one as often as experts before falling behind. Memories models write across runs do not improve them and are often wrong and overly cautious.
Interpretation
Long-horizon strategic reasoning now has a benchmark that runs a full episode, rather than short, episodic tasks. Most agentic benchmarks are short and episodic, and performance on long tasks is hard to dissect into stored knowledge, reasoning and luck. Twenty large language models played complete runs at the game's highest difficulty; frontier models beat heuristic baselines and matched the average human player while staying far behind experts.
What separates strong models from weak is how well they play each fight, not how they open. The gap widens as the run lengthens: the strongest models win the first act as often as experts, then fall behind. All models misjudge risk over a run, preserving health rather than spending it to grow stronger.
A comprehensive, self-consistent renaming of every game term shows part of their performance rests on prior knowledge of the game. Models do not improve from the memories they write across runs, which are often wrong and overly cautious. The harness is the first to give a model the complete original game, and it is compared with existing harnesses.
Perspective
The benchmark targets researchers and engineers studying long-horizon strategic reasoning in agents, using complete runs of Slay the Spire at the highest difficulty across 20 large language models, compared with heuristic baselines and human player levels.
Conclusions come from a single commercial deck-builder; memories models write across runs are often wrong and overly cautious and bring no improvement; the widening long-horizon gap is reported at the summary level, and its mechanism remains to be laid out.