Multimodal generation drops joint training: SnB splits one diffusion trajectory and reuses pretrained models
Synopsis
Multimodal generation has largely relied on jointly training a shared latent variable from scratch, which scales poorly to high-resolution images. SnB instead couples separately pretrained unimodal diffusion models at sampling time: the modalities share one reverse trajectory up to a split point, then a bridge module routes each into its own branch. On CelebAMask-HQ, SnB reaches the lowest FID across all reported shared-fraction settings while keeping strong coherence; on PolyMNIST it trades wins with top-down methods.
Interpretation
Multimodal generation no longer has to start from a jointly trained shared latent variable; separately trained unimodal diffusion models can be stitched together at sampling time to produce corresponding visual data on demand. Earlier top-down methods depend on joint training and restrictive joint-posterior assumptions such as product-of-experts or mixture-of-experts, and are limited by latent bottlenecks and a coherence-quality trade-off. In joint face-mask generation on CelebAMask-HQ, SnB attains the lowest FID at every reported shared-fraction setting while maintaining strong coherence, outperforming both VAE-based and diffusion-based top-down baselines.
The procedure denoises two modalities along one shared reverse diffusion trajectory, then lets each continue under its own model after the split time step, with a bridge module mapping the noisy state from one side to the other. The bridge is a time-conditioned U-Net trained with a reconstruction loss, and noise is re-injected after the mapping according to the diffusion scheduler so the output is not overly smooth and mismatched to the current noise level. On PolyMNIST the simplest identity bridge already balances coherence and quality in unconditional generation; on CelebAMask-HQ removing the learnable bridge sharply lowers coherence while FID stays around 60.
The same sampling mechanism can call large foundation diffusion models directly for interleaved co-generation, image editing, and style coupling and decoupling, without extra joint training. On PIE-Bench++ the methods have similar FID and CLIP scores, and the difference shows up mainly in pair consistency. Across four backbones, Flux, SDXL, SD 2.1 and SD 1.5, SnB's pair-consistency metrics beat both independent generation and shared-noise controls.
Perspective
The demonstrated scope covers five-modality digit images in PolyMNIST, 512×512 faces with 15-class masks in CelebAMask-HQ, interleaved co-generation and editing on PIE-Bench++, and zero-shot MRI contrast synthesis on ROCO across brain, joint and spine anatomy groups; it suits research and engineering readers interested in multimodal generation and diffusion-model reuse.
Choosing the bridge mechanism and split point requires empirical tuning by modality gap and data complexity, and the initial trajectory's modality affects results, with face-first outperforming mask-first; the method also assumes a usable pretrained unimodal model for each modality.