Encoder-free speech LLM pretrained on speaker-identity and emotion metadata beats a randomly initialized encoder baseline under matched data
The work proposes metadata-supervised pretraining (MSP), which uses speech attributes such as speaker identity and emotion, together with speaker-aware utterance composition (SAUC) and random span masking, to pretrain an encoder-free speech LLM; on joint ASR and speaker diarization in multi-speaker conversations, the encoder-free model outperforms its randomly initialized encoder-based counterpart under matched training-data conditions, and with limited metadata-annotated data it is competitive with models using speech encoders pretrained on substantially larger corpora, outperforming them in several settings.