Skip to main content
Search

# Multimodal Generation

2 related articles

Subscribe to updates
  1. arXiv

    Multimodal generation drops joint training: SnB splits one diffusion trajectory and reuses pretrained models

    Multimodal generation has largely relied on jointly training a shared latent variable from scratch, which scales poorly to high-resolution images. SnB instead couples separately pretrained unimodal diffusion models at sampling time: the modalities share one reverse trajectory up to a split point, then a bridge module routes each into its own branch. On CelebAMask-HQ, SnB reaches the lowest FID across all reported shared-fraction settings while keeping strong coherence; on PolyMNIST it trades wins with top-down methods.

  2. arXiv

    One bitstream diffusion recipe spans images, speech and proteins: 3.5% word error in joint speech-text, but two fold-level protein figures disagree

    Continuous bitstream diffusion writes images, speech and proteins as binary codes and, with one training and sampling protocol, reaches 3.5% word error rate in joint speech-text generation, while the fold-level agreement of generated protein sequence-structure pairs is reported as 75% in the results section and 69% in the conclusion.

You’re all caught up