# LLM research agents

In simulated worlds with a planted, known signal, Claude Opus 5.5, GPT-6.1 Sol, GPT-6.1 Terra and Qwen3-32B were asked whether a forecasting model beats a static benchmark. After Gaussian shocks removed a resampling artifact, the three commercial models claimed predictability in 34%, 16% and 44% of 110 no-signal worlds, all above the nominal 5%; once false-positive rates are matched, an in-sample test with a multiple-testing adjustment detects weak signals about as well as the best agent.

In simulated worlds with a planted, known signal, Claude Opus 5.5, GPT-6.1 Sol, GPT-6.1 Terra and Qwen3-32B were asked whether a forecasting model beats a static benchmark. After Gaussian shocks removed a resampling artifact, the three commercial models claimed predictability in 34%, 16% and 44% of 110 no-signal worlds, all above the nominal 5%; once false-positive rates are matched, an in-sample test with a multiple-testing adjustment detects weak signals about as well as the best agent.

AutoTavern coordinates specialised LLM research agents through a shared typed research memory and five verification gates, and in 20 days they built TavernGym, a full-content reproduction of Hearthstone Battlegrounds, plus 23 versions of TavernAgent. The agent plays ranked games on the live ladder by itself, with its rating rising from 6,000 to about 15,000; on the public leaderboard's measure its last 20 records average a placement of 1.60 with 80% first places, against 2.34 and 59% for the server's ten highest-rated players together. Asked to make the same decisions directly, frontier LLMs answer at most 63% of play tasks, while TavernAgent answers 91%.

AutoTavern coordinates specialised LLM research agents through a shared typed research memory and five verification gates, and in 20 days they built TavernGym, a full-content reproduction of Hearthstone Battlegrounds, plus 23 versions of TavernAgent. The agent plays ranked games on the live ladder by itself, with its rating rising from 6,000 to about 15,000; on the public leaderboard's measure its last 20 records average a placement of 1.60 with 80% first places, against 2.34 and 59% for the server's ten highest-rated players together. Asked to make the same decisions directly, frontier LLMs answer at most 63% of play tasks, while TavernAgent answers 91%.