Agents play full Slay the Spire runs: top models match average humans, then fall behind experts as runs lengthen

Long-horizon strategic play by agentic models is now measured inside a complete commercial game, not only in short episodic tasks. Most agentic benchmarks are short and episodic, and performance on long tasks is hard to dissect into stored knowledge, reasoning and luck. Across 20 large language models at the game's highest difficulty, frontier models beat heuristic baselines and match the average human player but remain far behind experts.

What separates strong models from weak ones is how well they play each fight. All of them misjudge risk over a run, preserving health rather than spending it to grow stronger. The gap widens as the run lengthens: the strongest models win the first act as often as experts, then fall behind.

Part of the models' performance rests on prior knowledge of the game rather than on experience accumulated across runs. A comprehensive, self-consistent renaming of every game term isolates the contribution of that prior knowledge. Models do not improve from the memories they write across runs, which are often wrong and overly cautious.

The work is aimed at agent researchers studying long-horizon strategic reasoning, and its evaluation setting is complete Slay the Spire runs at the highest difficulty, with a comparison against existing harnesses. The material states no next experiment or transfer recipe, so what a reader can follow is the benchmark and its failure-locating tools.

The material confines its conclusions to Slay the Spire and the highest-difficulty setting, and notes that the memories models write across runs are often wrong and overly cautious. Those qualifications leave open whether the same phenomena appear in other long-horizon tasks.