Agents that never play the game built a Hearthstone AI that climbed the live ladder to a 1.60 average placement
Synopsis
AutoTavern coordinates specialised LLM research agents through a shared typed research memory and five verification gates, and in 20 days they built TavernGym, a full-content reproduction of Hearthstone Battlegrounds, plus 23 versions of TavernAgent. The agent plays ranked games on the live ladder by itself, with its rating rising from 6,000 to about 15,000; on the public leaderboard's measure its last 20 records average a placement of 1.60 with 80% first places, against 2.34 and 59% for the server's ten highest-rated players together. Asked to make the same decisions directly, frontier LLMs answer at most 63% of play tasks, while TavernAgent answers 91%.
Interpretation
A team of specialised LLM research agents can carry a real game problem from raw data to a system that holds its own against people in 20 days, with humans only setting goals and ruling on ambiguous mechanics. Prior evidence on LLM research agents came from benchmark tasks or paper reviews, and it remained open whether agents could reach a system measured in the real game; no faithful Battlegrounds simulator existed, and earlier AI work either advised human players or studied simplified versions. Over 20 days the agents registered about 10,000 experiments and delivered 23 agent versions that passed the delivery gate, while humans typed 418 messages, 0.6% of all messages.
The system the agents built plays far better than the agents do: TavernAgent barely notices the real-time constraint, while frontier LLMs answering directly are shut out of live play by latency. On the 368 tasks every system answers, the best direct LLM, GPT-6 Astra, answers 63% and the two Claude models 52% to 53%, against 91% for TavernAgent. TavernAgent answers a task in 0.03 s while LLM median task times run 1.4 to 26 s; within the one-second budget TavernAgent keeps 90%, DeepSeek-V4.1-Flash 26% and Jev 37%, and every other LLM falls to 0%.
Fidelity of the training world, not a stronger learner, carries most of the gain: the learner faces opponent libraries recalibrated turn by turn to the opponents it actually met in live games. From the same checkpoint and budget, uncalibrated opponent libraries give a 20.9% first-place rate on the main exam and 10.3% on the harder one, while reality-calibrated libraries give 27.2% and 13.5%. All 23 delivered versions were re-measured on one simulator build, one exam and three unseen seeds, with the first-place rate rising from 1.9% to 41.5% and average placement from 3.40 to 2.33.
Perspective
The demonstrated scope is one game, one patch (36.6.3) and solo-mode games on the authors' own account, with training against calibrated opponent libraries; the ladder record covers 392 games on days 5 to 18 of the project. TavernBench and TavernGym are to be open-sourced at https://github.com/MCG-NJU/AutoTavern.
Rating rises only as fast as games are played, and the ladder ran for two weeks; the authors compare players through a public leaderboard tracker that estimates a placement for each rating update, which is not a count of games. Of 388 games with a valid placement, 96 carry client-failure flags that measure the client rather than the agent, and the authors note the agent's opponents are library seats rather than people, so the growth comparison describes development rather than head-to-head strength.