CARDIAC-FM: A Generalizable Multimodal Foundation Model Integrating ECG and Cardiac MRI
Synopsis
This study developed CARDIAC-FM, a multimodal foundation model that integrates 12-lead ECG and cardiac MRI through self-supervised representation learning and cross-modal contrastive alignment, pretrained on 57,609 paired samples from UK Biobank, improving prediction of incident atrial fibrillation and heart failure, generalizing zero-shot to the Cardiovascular Health Study and the Multi-Ethnic Study of Atherosclerosis, and transferring to cardiac MRI phenotype prediction, time-to-event modelling, and additional outcomes including myocardial infarction, ischaemic stroke, cardiovascular death, and all-cause mortality.
raw ECG and cardiac MRI inputs (Fig. 1). The cardiac MRI encoder is a new spatiotemporal, multi-view masked autoencoder, pretrained from scratch on raw cine MRI without phenotype or outcome supervision. The encoder jointly processes short-axis, four-chamber and two-chamber views using spatiotemporal masking and a Vision Transformer29 architecture. The resulting cardiac MRI representation is aligned with the pretrained ECG representation in a shared latent space using a symmetric InfoNCE contrastive loss30. The model was trained on 34,596 samples with paired ECG and MRI data. For downstream outcome prediction, we excluded participants with prevalent disease at baseline, added a prediction layer to the learned representations, and fine-tuned the model for 20 epochs using supervised learning. CARDIAC-FM supports two modality configurations, ECG-only and ECG+MRI; either prediction can additionally be combined with a clinical risk score through late fusion. Full methodological details are provided in Methods.
medRxiv · Page 5Interpretation
It introduces and evaluates a multimodal foundation model integrating ECG and cardiac MRI, whose cardiac MRI encoder is a self-supervised spatiotemporal, multi-view masked autoencoder pretrained from scratch on raw cine MRI without phenotype or outcome supervision. Relative to prior models that are mostly single-modality, task-specific, or focused on prevalent disease, this work combines modality-specific self-supervised pretraining with cross-modal contrastive alignment and supports both ECG-only and ECG+MRI inference configurations. Pretrained on 57,609 paired high-quality samples from UK Biobank, with model code and weights publicly released; compared against four contemporary ECG AI models (ECG-FM, ECGFounder, DeepECG-SL, DeepECG-SSL) and the CHARGE-AF and PREVENT-HF clinical risk scores under a common fine-tuning protocol with bootstrap confidence intervals.
Multimodal pretraining improved prediction even when only ECG was available at inference, adding cardiac MRI further improved performance, and combining with clinical risk scores improved discrimination further. This indicates that the ECG representation can absorb information captured by MRI, and that ECG, MRI, and traditional risk factors provide complementary prognostic information. In the UK Biobank test set, atrial fibrillation ECG-only AUROC was 0.769 (95% CI 0.737–0.801) with AUPRC 0.291, and ECG+MRI AUROC was 0.816 with AUPRC 0.337; heart failure ECG-only AUROC was 0.817 with AUPRC 0.226, and ECG+MRI AUROC was 0.830 with AUPRC 0.265; combining with risk scores raised heart failure AUROC from 0.751 to 0.792 in CHS and from 0.789 to 0.826 in MESA.
Trained in UK Biobank, the model transferred zero-shot to two independent US prospective cohorts, CHS and MESA, and performed well in time-to-event modelling and additional cardiovascular outcomes. Relative to models developed in a single or highly selected cohort, this work demonstrates transfer across populations, outcomes, and prediction task formulations. In zero-shot evaluation, CARDIAC-FM significantly outperformed three of four comparator ECG AI models for both atrial fibrillation and heart failure in AUROC and AUPRC in both CHS and MESA; UK Biobank time-to-event C-indices were 0.740 for atrial fibrillation and 0.786 for heart failure; after fine-tuning with 10% or 20% of each cohort, it generally achieved the highest AUROC and AUPRC for myocardial infarction, ischaemic stroke, cardiovascular death, and all-cause mortality, apart from ischaemic stroke in MESA.
ECG could predict cardiac MRI structural and functional phenotypes, and these predicted phenotypes preserved prognostic associations similar to directly measured phenotypes. This suggests the multimodally pretrained representation can recover clinically relevant cardiac MRI-related information from ECG, offering substitute information when MRI is unavailable. ECG-predicted values correlated with direct imaging measures at Pearson r=0.45–0.79, with higher correlations for structural measures (LVM r=0.79, LVEDV r=0.72, LVESV r=0.71) than functional indices (LVEF r=0.49, LAEF r=0.45); in MESA, predicted and measured phenotypes were concordant in direction across 28 phenotype–outcome associations with a Pearson correlation of log-HRs of r=0.93 and overlapping 95% confidence intervals in 23 of 28.
Perspective
The framework is intended for cardiovascular risk prediction settings where ECG serves as a scalable screening signal and cardiac MRI provides rich information during development, accommodating clinical and population-screening environments with differing data availability; the ECG-only configuration suits population-scale applications where MRI is impractical or unavailable, while the ECG+MRI configuration can additionally exploit subject-specific imaging information when available; late fusion with clinical risk scores such as CHARGE-AF or PREVENT-HF can further improve discrimination.
The model was pretrained predominantly in UK Biobank imaging participants, and broader evaluation across healthcare systems, geographic regions, and clinical populations remains to be conducted; zero-shot evaluation showed transfer of discrimination, but absolute-risk calibration shifted across cohorts and improved after cross-fitted Platt recalibration, suggesting local recalibration may be needed before clinical deployment; ECG-derived cardiac MRI phenotypes only moderately approximate directly measured phenotypes and should not be interpreted as replacements for cardiac MRI; improvements in discrimination, risk stratification, and decision-curve net benefit do not by themselves establish clinical effectiveness, and prospective studies are needed to determine whether the model improves clinical decisions and patient outcomes.