Scott C. Lowe

Agentic large language models handle complex tasks, yet long-horizon strategic reasoning lacks a dissectable benchmark. SpireBench has 20 LLMs play the complete commercial deck-builder Slay the Spire at its highest difficulty: frontier models beat heuristic baselines and match the average human player but remain far behind experts, and the gap widens as runs lengthen, with the strongest models winning act one as often as experts before falling behind. Memories models write across runs do not improve them and are often wrong and overly cautious.

Agentic large language models handle complex tasks, yet long-horizon strategic reasoning lacks a dissectable benchmark. SpireBench has 20 LLMs play the complete commercial deck-builder Slay the Spire at its highest difficulty: frontier models beat heuristic baselines and match the average human player but remain far behind experts, and the gap widens as runs lengthen, with the strongest models winning act one as often as experts before falling behind. Memories models write across runs do not improve them and are often wrong and overly cautious.