One bitstream diffusion recipe spans images, speech and proteins: 3.5% word error in joint speech-text, but two fold-level protein figures disagree

A single denoising interface now handles image-text, speech-text and protein sequence-structure pairs, with joint and conditional generation sharing one set of weights. Earlier multimodal systems bridged modalities with modality-specific encoders, latent spaces, objectives or prediction interfaces, so each new pairing added another interface to design and calibrate. On MNIST-Sum, one 63.0M-parameter model serves all three directions: image-to-text exact match is 97.56%, text-to-image places all four digits in 99.95% of samples, and joint generation from noise is 99.46% mutually consistent.

Each modality is first written as bits and placed in one fixed-layout bitstream, and the denoiser sees only bits, not modality identity. Discrete symbols become binary codes directly, while continuous observations pass through a frozen, modality-specific tokenizer before sharing one integer code space with text codes. Speech and text concatenate text, speaker and speech segments whose vocabularies fit into a common 18-bit word; protein residues align five amino-acid bits with thirteen structure-code bits.

Training hides part of the bitstream and scores only the hidden bits, so which bits are observed determines what gets generated. Joint generation, one-directional conditional generation and continuation no longer need separate losses or sampling procedures; only the set of observed positions changes. A speech-text model trained from scratch on LibriTTS and MLS-English co-generates text and speech at 3.5% word error rate between generated text and automatic transcriptions of generated speech; a protein model trained from scratch on 216,237 PDB/AFDB-SwissProt pairs gives, over 100 generated pairs after one ProteinMPNN redesign and ESMFold refold, a mean self-consistency TM-score of 0.899 with 75% fold-level and 36% coordinate-level success in the results section, while the conclusion reports 69% fold-level agreement at scTM about 0.9 and 41% at scRMSD 2 Å, so the two sections disagree.

The protocol is validated on three pairings: image-text, speech-text and protein sequence-structure, with denoisers from 63M to 633M parameters, all trained from scratch. The speech experiments use the read-speech LibriTTS and MLS-English corpora, and the protein experiments use 216,237 PDB/AFDB-SwissProt pairs with 20% sequence replay. It is aimed at researchers and engineers interested in unifying cross-modal generation interfaces; the source proposes text-pretrained initialization, larger backbones, broader data and better continuous-modality tokenizers as next steps within the same protocol.

Speech recognition is the overall weakest direction, with word-level errors dominated by substitutions, which the authors tie to the absence of text pretraining. Speech continuation struggles to preserve thematic content, and speaker replication is limited by the similarity ceiling of StableCodec reconstructions and by the audio decoder not being conditioned on the speaker. For proteins, fold-level success exceeds coordinate-level success, and coordinate-level success falls as sequences get longer; the authors note this points to geometric precision needing dedicated protein training, while establishing biological function is beyond the scope of the work. The fold-level success rate for joint protein generation is given as two different values in the results section and the conclusion, and this report does not choose between them.