Handing forecast verdicts to AI agents: in no-signal worlds Opus, Sol and Terra still claim predictability 34%, 16% and 44% of the time
Synopsis
In simulated worlds with a planted, known signal, Claude Opus 5.5, GPT-6.1 Sol, GPT-6.1 Terra and Qwen3-32B were asked whether a forecasting model beats a static benchmark. After Gaussian shocks removed a resampling artifact, the three commercial models claimed predictability in 34%, 16% and 44% of 110 no-signal worlds, all above the nominal 5%; once false-positive rates are matched, an in-sample test with a multiple-testing adjustment detects weak signals about as well as the best agent.
Interpretation
The study scores an LLM research agent as a statistical test rather than judging whether its conclusion sounds plausible. With real data the truth is unknown, so a skeptical verdict could not be separated from a weak search and an optimistic one from data mining. The data-generating process of each simulated world is controlled, with population R-squared set to 0, 0.25, 0.5, 1, 2 or 4% per month, so every verdict can be scored against a known truth.
The agent claims predictability more often than the pre-specified out-of-sample test in weak-signal worlds, mainly because it accepts weaker evidence. The pre-specified baseline decides by a one-sided Clark–West test at the 5% level using pseudo-out-of-sample forecasts over the second half of the training window, which makes it conservative. In the 240 paired weak-signal worlds the agent claims in 38% and the primary baseline in 8%, and all 72 discordant pairs favor the agent.
Once false-positive rates are matched, the agent's search advantage largely disappears. The agent claims in 15% of linear null worlds, and each baseline's significance level is set so that it claims at the same rate before comparison. At a 1% signal level the agent detects 61%, the matched HAC-Holm test 59%, IVX 54% and an oracle that knows the true model 79%.
After removing a repeated-value artifact from bootstrap resampling, all three commercial models still over-claim predictability in no-signal worlds. The first two studies drew return shocks from a finite pool of HML residuals, so values repeat exactly and a lookup of past returns can forecast a world with no signal; Terra exploited this in 65 runs. In the third study with Gaussian shocks, Opus claims in 34%, Sol in 16% and Terra in 44% of 110 no-signal worlds, each with p below 0.001 against 5%.
Perspective
The conclusions rest on controlled simulated worlds: linear worlds contain one planted predictor and seven distractors with no predictive power, alongside trap worlds, worlds with a signal injected into real factors, and a setting of 700 training months and 360 test months. The intended readers are empirical researchers who use research agents to judge predictability, and the authors recommend that the agent search while the decision rule and holdout stay outside its reach.
The authors state that external validity is limited: three commercial models from two developers and one open-weight model are studied under one agent configuration, one tool set, one brief and a fixed reasoning effort per model, and the same-named high reasoning settings are not calibrated against each other. The worlds are stylized, and several true predictors, drifting or breaking slopes and frequencies other than monthly remain untested; the Gaussian worlds remove the artifact but also the fat tails and volatility clustering of real returns. The false-positive rate in linear null worlds rests on 40 worlds, so the matched comparison is approximate.