Three LLMs locked predictions before ESC 2026: from registered design alone, the best AUC for separating positive from negative trials was 0.80

Large language models can assign a discriminating probability to whether a cardiovascular trial will meet its primary endpoint, using only design information available at registration and before any result is public. Earlier tests of model trial prediction were retrospective and could not exclude recall of published answers; oncologists predicting trials in their own subspecialty reached AUCs of 0.52 and 0.43, and principal investigators predicting their own trials reached 0.76. Of 48 scorable trials, 28 were positive; AUCs were 0.80 (95% CI 0.66–0.92) for Claude, 0.79 (0.66–0.91) for GPT and 0.74 (0.59–0.87) for Gemini, with P<0.001, P<0.001 and P=0.004, and accuracies of 73%, 71% and 67%.

Predictions were made and registered before the congress opened, and the models had no web access, so the results cannot come from recalling published answers. Each model received the same input through its API: the trial title, a structured design summary compiled by the investigators from the public registration, and investigator-written definitions of a positive result and the effect measure, answering three times under default sampling with the median scored. The protocol, design summaries, locked predictions and adjudication rules were registered on the Open Science Framework on 22 August 2026, before the congress; predictions were generated on 21–22 August, and results were adjudicated from simultaneous publications (19 trials) or presenters' slides (31).

The models also gave point estimates of effect size with 80% prediction intervals, and interval coverage differed by model. Among 26 trials whose effect was reported exactly as locked, the direction was correct in 88% of predictions, and half of the ratio predictions fell within about 15% of the observed value. The 80% prediction intervals contained the observed effect in 19/26 (73.1%) for Claude, 22/26 (84.6%) for GPT and 15/26 (57.7%) for Gemini.

The demonstrated scope is the 50 trials of the ESC 2026 Late-Breaking Science programme, restricted to binary met-or-not-met primary endpoints and to inputs consisting of the title and an investigator-compiled design summary; the authors state that a dedicated test is needed to show whether such predictions help trial design and whether the finding holds for later models.

A single investigator who had seen the predictions adjudicated results under registered rules; two trials were matched to the wrong registry record; six trials were scored although the presented endpoint or timepoint differed from the locked one; only half of the trials reported the effect measure exactly as locked; the models tended to be wrong on the same trials, so agreement between them is not independent confirmation, and the 80% intervals did not reach 80% coverage in every model.