One clinical foundation model, two ways to predict: representations win on risk, generation wins on same-day orders

A single pretrained model can be used two ways, and which one wins depends on the clinical task: compressing a history into a numerical summary and handing it to a task-specific predictor is better for estimating the multi-year risk of a prespecified outcome, while letting the model extend the history by generating event sequences is better for predicting which diagnoses, prescriptions, and procedures will be recorded together on the next clinical date. Earlier work used patient representations to train outcome predictors, as in CLMBR and MOTOR, or generated event sequences autoregressively, as in Delphi-2M, Curiosity, ETHOS, and Foresight, but the relative strength of the two uses had not been compared within one model holding patients, observed histories, and outcome definitions fixed. Among 10,000 test patients, representation-based prediction exceeded generation-based estimates by 0.098–0.115 observation-weighted AUROC for five-year chronic kidney disease, fatty liver, and diabetes, with paired 95% intervals excluding zero; among 1,862 test patients with an observed target, generation with 50 trajectories per patient reached a Recall@10 of 71.00% versus 55.68% for representation-based prediction, a paired improvement of 15.31 percentage points (95% CI, 13.92–16.79).

The model orders each patient's records by date as clinical tokens, replaces positional encoding with continuous age, and decides separately whether the next event falls on the current date before deciding how long to wait. Delphi-2M derives the waiting time from the event logits, and ETHOS and Curiosity insert discrete time-interval tokens between events and omit them when events are closer together, whereas FERMAT uses a same-day head to predict whether the next event occurs on the current date and a separate time head that models the waiting time in continuous time. Pretraining used longitudinal records in the OMOP Common Data Model from a single hospital covering October 2004 to December 2023, with 3,773,556 patients and 1,238,329,877 tokenized events split by patient into 2,642,237 training, 565,534 validation, and 565,785 test patients; the model is a 65.22-million-parameter autoregressive transformer with a 512-token context, trained for 100,000 event-prediction steps followed by 100,000 steps with an additional time-prediction loss weighted by 0.3.

The comparison covers two tasks: one-to-five-year incidence risk for 31 diseases in a 10,000-patient test cohort, and the set of diagnoses, prescriptions, and procedures recorded on the first clinical date after the observed history in 1,862 test patients. Both uses share one pretrained model, the same patients, the same observed histories, and the same outcome definitions, with the same adjustment for incomplete follow-up. The authors propose testing the task-dependent pattern in other institutions and linked claims data, and exploring ways to combine representations with generated trajectories.

Data from a single hospital may limit generalizability to populations and healthcare settings with different clinical and documentation practices. Records may not fully capture patients' clinical histories: the first recorded diagnosis date may differ from disease onset or initial diagnosis, and events occurring outside the source hospital may be missing. Outcomes were defined by a single first recorded diagnosis, without repeated diagnoses, confirming prescriptions or procedures, or registry linkage, and billing practices can inflate recorded diagnoses. Observation weighting assumes that, given age, sex, utilization, and prior disease indicators, whether an outcome is observed is unrelated to the outcome; this fails if patients leave the source hospital for reasons those variables do not capture, and diagnoses made elsewhere are missed regardless. The authors also note that supervised fine-tuning, for example with low-rank adaptation, was not evaluated and could change the relative performance in either task.