# Cross-Modal Alignment

Shanghai Jiao Tong University and collaborators assembled a roughly 47,000-hour multisource pool and extended a pretrained video model's language–video priors into language–video–action learning, yielding the world action model SWAM. On the 50 manipulation tasks of RoboTwin 2.0, SWAM reaches 61.56% success under the randomized Hard setting, 12.86 percentage points above the strongest baseline OpenWAM, and 88.10% under the clean Easy setting, slightly below OpenWAM's 89.30%. On a real bimanual platform, its three-task average Hard success rate is 40.00% against OpenWAM's 23.33%.

Shanghai Jiao Tong University and collaborators assembled a roughly 47,000-hour multisource pool and extended a pretrained video model's language–video priors into language–video–action learning, yielding the world action model SWAM. On the 50 manipulation tasks of RoboTwin 2.0, SWAM reaches 61.56% success under the randomized Hard setting, 12.86 percentage points above the strongest baseline OpenWAM, and 88.10% under the clean Easy setting, slightly below OpenWAM's 89.30%. On a real bimanual platform, its three-task average Hard success rate is 40.00% against OpenWAM's 23.33%.

Medical CLIP has mainly aligned high-level semantic features, leaving texture, entity order and attributes poorly captured and limiting fine-grained tasks. Deco-CLIP splits both image and text into two branches: one ViT pretrained by CT-CLIP for semantics, one MAE-pretrained ViT for texture, with a text encoder that forks in its last three layers, shares keys and values, and uses distinct queries, plus threshold-adaptive sample mining and a regularized attention mechanism. On CT-RATE zero-shot tasks, multi-abnormality detection accuracy improves by 10.2% and report-to-volume retrieval recall by 16.2%.

Medical CLIP has mainly aligned high-level semantic features, leaving texture, entity order and attributes poorly captured and limiting fine-grained tasks. Deco-CLIP splits both image and text into two branches: one ViT pretrained by CT-CLIP for semantics, one MAE-pretrained ViT for texture, with a text encoder that forks in its last three layers, shares keys and values, and uses distinct queries, plus threshold-adaptive sample mining and a regularized attention mechanism. On CT-RATE zero-shot tasks, multi-abnormality detection accuracy improves by 10.2% and report-to-volume retrieval recall by 16.2%.

Soft robotic grasping lacks multimodal data that coordinates vision, depth, touch, inertial measurement and text, and existing alignment methods apply one strength to every modality pair. SomaAlign first estimates each modality's relative reliability from masked reconstruction, then uses it to set a pair-specific contrastive temperature, and in a third stage tightens intra-sample cross-modal consistency with conflict-aware constraints. Across 30 directed cross-modal retrieval tasks on the Soma dataset the mean Recall@10 is 90.66%, and on the independently collected Soma-Success set, excluded from representation learning, frozen encoders with lightweight probes reach 94.59% post-hoc grasp-outcome classification accuracy.

Soft robotic grasping lacks multimodal data that coordinates vision, depth, touch, inertial measurement and text, and existing alignment methods apply one strength to every modality pair. SomaAlign first estimates each modality's relative reliability from masked reconstruction, then uses it to set a pair-specific contrastive temperature, and in a third stage tightens intra-sample cross-modal consistency with conflict-aware constraints. Across 30 directed cross-modal retrieval tasks on the Soma dataset the mean Recall@10 is 90.66%, and on the independently collected Soma-Success set, excluded from representation learning, frozen encoders with lightweight probes reach 94.59% post-hoc grasp-outcome classification accuracy.

The work introduces PEACE, the first joint embedding that maps Faust effect code and processed audio into one space: the audio side reuses AFx-Rep while the code side is either a fine-tuned T5 or BoxGraph, a message-passing graph neural network over the Faust compiler's Box API intermediate representation, trained with SLAP's non-contrastive BYOL objective; across 200K training pairs, 21 Faust effects and chains of length 1-3, the two code encoders roughly tie on mixed-length chain retrieval, T5 leads slightly on single-effect galleries and longer chains, BoxGraph is stronger at parameter-masked chain-topology retrieval (17.4% versus 12.4% R@1) and reaches 53.5% R@10 on a reverb benchmark built from professional plugins unseen in training, above pretrained AFx-Rep's 28.3%.

The work introduces PEACE, the first joint embedding that maps Faust effect code and processed audio into one space: the audio side reuses AFx-Rep while the code side is either a fine-tuned T5 or BoxGraph, a message-passing graph neural network over the Faust compiler's Box API intermediate representation, trained with SLAP's non-contrastive BYOL objective; across 200K training pairs, 21 Faust effects and chains of length 1-3, the two code encoders roughly tie on mixed-length chain retrieval, T5 leads slightly on single-effect galleries and longer chains, BoxGraph is stronger at parameter-masked chain-topology retrieval (17.4% versus 12.4% R@1) and reaches 53.5% R@10 on a reverb benchmark built from professional plugins unseen in training, above pretrained AFx-Rep's 28.3%.

The work proposes Relation-Conditioned Multimodal Learning (RCML), a framework that treats natural-language relation descriptions as explicit conditions of multimodal representation learning by constructing relation-aware training pairs, introducing a relation-conditioned module to adapt embeddings to relation semantics, and employing a unified contrastive objective that jointly models cross-modal alignment and relation-induced inter-sample structure, so the same sample can be represented differently under different relational contexts; experiments on multiple datasets show RCML consistently outperforms strong baselines on retrieval and classification tasks in zero-shot, fine-tuned, and out-of-domain settings.

The work proposes Relation-Conditioned Multimodal Learning (RCML), a framework that treats natural-language relation descriptions as explicit conditions of multimodal representation learning by constructing relation-aware training pairs, introducing a relation-conditioned module to adapt embeddings to relation semantics, and employing a unified contrastive objective that jointly models cross-modal alignment and relation-induced inter-sample structure, so the same sample can be represented differently under different relational contexts; experiments on multiple datasets show RCML consistently outperforms strong baselines on retrieval and classification tasks in zero-shot, fine-tuned, and out-of-domain settings.

This study developed CARDIAC-FM, a multimodal foundation model that integrates 12-lead ECG and cardiac MRI through self-supervised representation learning and cross-modal contrastive alignment, pretrained on 57,609 paired samples from UK Biobank, improving prediction of incident atrial fibrillation and heart failure, generalizing zero-shot to the Cardiovascular Health Study and the Multi-Ethnic Study of Atherosclerosis, and transferring to cardiac MRI phenotype prediction, time-to-event modelling, and additional outcomes including myocardial infarction, ischaemic stroke, cardiovascular death, and all-cause mortality.

This study developed CARDIAC-FM, a multimodal foundation model that integrates 12-lead ECG and cardiac MRI through self-supervised representation learning and cross-modal contrastive alignment, pretrained on 57,609 paired samples from UK Biobank, improving prediction of incident atrial fibrillation and heart failure, generalizing zero-shot to the Cardiovascular Health Study and the Multi-Ethnic Study of Atherosclerosis, and transferring to cardiac MRI phenotype prediction, time-to-event modelling, and additional outcomes including myocardial infarction, ischaemic stroke, cardiovascular death, and all-cause mortality.