# Multimodal Deep Learning

Swiss forest composition was previously shown as one dominant species per pixel, erasing the share of accompanying species. A deep-learning model anchored on 0.1 m aerial RGB imagery and combined with airborne laser scanning, annual AlphaEarth satellite embeddings and climate variables was trained on basal-area observations from 3,992 Swiss National Forest Inventory sites to output basal-area shares for ten main tree species and an Other class on a 30 m grid. Spatial cross-validation gave a macro coefficient of determination of 0.485; on forest-reserve data not used in model development, the macro value was 0.552 at 30 m pixel support and 0.733 at plot support, with dominant-class identification in 73.2% of pixels and 81.1% of plots.

Swiss forest composition was previously shown as one dominant species per pixel, erasing the share of accompanying species. A deep-learning model anchored on 0.1 m aerial RGB imagery and combined with airborne laser scanning, annual AlphaEarth satellite embeddings and climate variables was trained on basal-area observations from 3,992 Swiss National Forest Inventory sites to output basal-area shares for ten main tree species and an Other class on a 30 m grid. Spatial cross-validation gave a macro coefficient of determination of 0.485; on forest-reserve data not used in model development, the macro value was 0.552 at 30 m pixel support and 0.733 at plot support, with dominant-class identification in 73.2% of pixels and 81.1% of plots.

Using 203,434 ICU admissions from 2001 to 2022 across more than 200 hospitals in the MIMIC-III, MIMIC-IV, eICU, and HiRID databases, the study developed and externally validated a multimodal deep-learning model that predicts subsequent inpatient mortality from time-invariant variables, time-variant variables, clinical notes, and chest x-ray images available within the first 24 h of ICU admission; with structured data alone the model reached an AUROC of 0.92 (95% CI, 0.90 to 0.93), an AUPRC of 0.53 (95% CI, 0.49 to 0.57), and a Brier score of 0.19 (95% CI, 0.18 to 0.20), external validation across eight eICU institutions yielded AUROCs of 0.84 to 0.92, and in the subgroup with both notes and imaging, adding text and images raised the AUROC modestly from 0.87 (95% CI, 0.85 to 0.89) to 0.

Using 203,434 ICU admissions from 2001 to 2022 across more than 200 hospitals in the MIMIC-III, MIMIC-IV, eICU, and HiRID databases, the study developed and externally validated a multimodal deep-learning model that predicts subsequent inpatient mortality from time-invariant variables, time-variant variables, clinical notes, and chest x-ray images available within the first 24 h of ICU admission; with structured data alone the model reached an AUROC of 0.92 (95% CI, 0.90 to 0.93), an AUPRC of 0.53 (95% CI, 0.49 to 0.57), and a Brier score of 0.19 (95% CI, 0.18 to 0.20), external validation across eight eICU institutions yielded AUROCs of 0.84 to 0.92, and in the subgroup with both notes and imaging, adding text and images raised the AUROC modestly from 0.87 (95% CI, 0.85 to 0.89) to 0.