# Multimodal Fusion

Multimodal cell foundation models usually freeze each modality's encoder and train only a fusion interface, yet whether fusion adds information beyond the best single modality has lacked a test. Researchers introduce the Synergistic Information Score (SIS), which measures accessible synergy as the gain of a fused representation over the strongest unimodal one under a linear probe. Across lung, thymus and breast spatial transcriptomics, unimodal baselines suffice in redundancy-dominated tasks, while non-spectral fusion reaches higher SIS in the resolution-mismatched thymus task.

Multimodal cell foundation models usually freeze each modality's encoder and train only a fusion interface, yet whether fusion adds information beyond the best single modality has lacked a test. Researchers introduce the Synergistic Information Score (SIS), which measures accessible synergy as the gain of a fused representation over the strongest unimodal one under a linear probe. Across lung, thymus and breast spatial transcriptomics, unimodal baselines suffice in redundancy-dominated tasks, while non-spectral fusion reaches higher SIS in the resolution-mismatched thymus task.

Tropical cyclone intensity forecasting has relied on best-track data or geostationary satellite imagery, yet similar cloud patterns can accompany intensification, steady state or weakening, and generic fusion strategies cannot tell them apart. TCFNet defines intensity trend from pressure gradients, uses it to select positive pairs for contrastive learning so the visual encoder learns cloud representations aligned with intensity change, then propagates states continuously to arbitrary lead times with neural ordinary differential equations. For 24-hour forecasts in the Western North Pacific and South Pacific, pressure RMSE is 9.52 hPa and 7.00 hPa respectively, and the critical success index for rapid intensification is 45.26%.

Tropical cyclone intensity forecasting has relied on best-track data or geostationary satellite imagery, yet similar cloud patterns can accompany intensification, steady state or weakening, and generic fusion strategies cannot tell them apart. TCFNet defines intensity trend from pressure gradients, uses it to select positive pairs for contrastive learning so the visual encoder learns cloud representations aligned with intensity change, then propagates states continuously to arbitrary lead times with neural ordinary differential equations. For 24-hour forecasts in the Western North Pacific and South Pacific, pressure RMSE is 9.52 hPa and 7.00 hPa respectively, and the critical success index for rapid intensification is 45.26%.

Structural MRI brain graphs no longer rely on a single predefined connection rule. MVMGNN treats brain regions as nodes carrying radiomics features, builds a spatial-neighbor graph and a feature-similarity graph, filters redundant features and weak connections inside each view with a joint node-edge mask, and fuses the two views through patient-level gated fusion. On the ADNI cohort, AD versus CN accuracy rises from 82.06% for the best comparison method, RSEA-MVGNN, to 89.90%, and F1 from 75.02% to 86.24%; gains on the harder AD-MCI and CN-MCI tasks are smaller.

Structural MRI brain graphs no longer rely on a single predefined connection rule. MVMGNN treats brain regions as nodes carrying radiomics features, builds a spatial-neighbor graph and a feature-similarity graph, filters redundant features and weak connections inside each view with a joint node-edge mask, and fuses the two views through patient-level gated fusion. On the ADNI cohort, AD versus CN accuracy rises from 82.06% for the best comparison method, RSEA-MVGNN, to 89.90%, and F1 from 75.02% to 86.24%; gains on the harder AD-MCI and CN-MCI tasks are smaller.

Recognition of psychological stress in dialogue has mostly treated each utterance as an isolated fragment and concatenated modality features at the final layer, which cannot capture how emotion accumulates and shifts across turns. BERT-EmoShiftNet pairs BERT's multi-level semantic encoding with EmoShiftNet's emotion-transition modelling, aligns text, speech and vision through a cross-modal Transformer, and unifies them with cross-modal contrastive learning in a shared embedding space. On a dialogue-level split of a self-built dataset, full-modal five-class accuracy is 85.0% (95% CI [83.2%, 86.8%]), emotion-transition prediction accuracy is 76.2%, and mean detection latency is 13.4 s.

Recognition of psychological stress in dialogue has mostly treated each utterance as an isolated fragment and concatenated modality features at the final layer, which cannot capture how emotion accumulates and shifts across turns. BERT-EmoShiftNet pairs BERT's multi-level semantic encoding with EmoShiftNet's emotion-transition modelling, aligns text, speech and vision through a cross-modal Transformer, and unifies them with cross-modal contrastive learning in a shared embedding space. On a dialogue-level split of a self-built dataset, full-modal five-class accuracy is 85.0% (95% CI [83.2%, 86.8%]), emotion-transition prediction accuracy is 76.2%, and mean detection latency is 13.4 s.

Visible and infrared images follow different imaging mechanisms, so direct concatenation or one-shot attention fusion can mistake background texture and thermal interference for complementary evidence. FCI-DETR splits detection into three linked stages: preserving modality-specific structure before fusion, calibrating cross-modal responses during fusion, and injecting infrared context by scale after fusion. It reaches 88.6%, 84.3%, 86.7% and 81.5% mAP50 on M3FD, DroneVehicle, VEDAI and FLIR, and beats a same-backbone RGB+IR RT-DETR baseline by 6.0 points on M3FD under a fixed protocol.

Visible and infrared images follow different imaging mechanisms, so direct concatenation or one-shot attention fusion can mistake background texture and thermal interference for complementary evidence. FCI-DETR splits detection into three linked stages: preserving modality-specific structure before fusion, calibrating cross-modal responses during fusion, and injecting infrared context by scale after fusion. It reaches 88.6%, 84.3%, 86.7% and 81.5% mAP50 on M3FD, DroneVehicle, VEDAI and FLIR, and beats a same-backbone RGB+IR RT-DETR baseline by 6.0 points on M3FD under a fixed protocol.

A Peking University-led team aligned routine single-lead ECG with physician-authored echocardiography reports to train AnyECG-Echo, a screening framework that resolves 13 fine-grained structural heart disease subtypes at once. Earlier AI-ECG models mostly required 12-lead input, were validated within a single health system, or needed per-condition manual labels. Across internal, temporal and geographically independent external cohorts totaling 25,222 patients, the framework reached high AUROC for high-impact subtypes such as reduced left ventricular systolic function, global heart enlargement and mitral stenosis, and retained usable discrimination in the external cohort.

A Peking University-led team aligned routine single-lead ECG with physician-authored echocardiography reports to train AnyECG-Echo, a screening framework that resolves 13 fine-grained structural heart disease subtypes at once. Earlier AI-ECG models mostly required 12-lead input, were validated within a single health system, or needed per-condition manual labels. Across internal, temporal and geographically independent external cohorts totaling 25,222 patients, the framework reached high AUROC for high-impact subtypes such as reduced left ventricular systolic function, global heart enlargement and mitral stenosis, and retained usable discrimination in the external cohort.

The study runs three connected analyses (alignment decoupling, attention routing and entropy, intrinsic dimensionality) plus causal intervention on representative concatenation and native multimodal LLMs, finding that concatenation models follow a text-first, vision-later fusion pathway while native models show earlier visual-textual co-adaptation and feature-space reorganization, offering a mechanistic view of multimodal fusion and architecture-aware diagnostics.

The study runs three connected analyses (alignment decoupling, attention routing and entropy, intrinsic dimensionality) plus causal intervention on representative concatenation and native multimodal LLMs, finding that concatenation models follow a text-first, vision-later fusion pathway while native models show earlier visual-textual co-adaptation and feature-space reorganization, offering a mechanistic view of multimodal fusion and architecture-aware diagnostics.