Unite-Audio jointly trains audio representation learning with latent flow matching, reaching competitive text-to-audio generation with a compact model

It proposes Unite-Audio, described by the authors as the first to jointly learn continuous audio representations and latent flow matching for text-to-audio generation. Most prior text-to-audio systems use a two-stage latent paradigm: an audio tokenizer is optimized for reconstruction and then frozen, after which a generative model is trained in that latent space; this work places representation learning and generative learning in one framework. Based on the abstract's positioning of the method against the two-stage paradigm; it is a method-level claim, and the given text provides no architectural details or comparison numbers.

By coupling reconstruction with self-supervised generative prediction, the generative objective directly shapes the latent space rather than treating it as a fixed intermediate representation. This departs from the convention in which the latent space is determined by reconstruction quality alone, orienting the representation toward generation. The abstract states this design as a mechanism and offers the observation that reconstruction-oriented representations may be suboptimal for generation as its motivation.

It employs Flow-GRPO post-training to improve text-conditioned generation. Beyond joint training, a post-training stage further optimizes text-conditioned generation quality. The abstract explicitly lists this post-training step but gives no configuration details or separate effect numbers in the provided text.

Experiments show competitive text-to-audio performance with a compact latent flow model, and ablation studies confirm the benefit of jointly learning the audio representation and generative model. It reaches competitive results at a compact model scale and attributes the performance gain to the joint-learning design choice through ablations. The abstract provides these concluding statements about experiments and ablations and links to audio samples; the given text contains no specific metrics, datasets, or numbers.

The work targets text-to-audio generation and applies to research and system-design settings that aim for competitive generation quality with a compact latent flow model; its central claim applies to text-to-audio systems built on the latent generative paradigm and suggests that later work can jointly optimize representation learning and the generative objective instead of freezing a tokenizer first. The abstract also links to audio samples so readers can listen to the generated results directly.

The given text is an abstract and contains no specific evaluation metrics, datasets, model scale, or ablation numbers, so it is not possible to judge against which baselines and under which conditions the 'competitive performance' holds; the independent contribution of Flow-GRPO post-training, the stability of joint learning across audio types, and how representative the linked audio samples are remain open questions for readers of the full paper.