Multi-talker transcription stops decoding each speaker alone: Pandemonium lets speakers see each other and reaches 14.7 average cpWER on meeting benchmarks
Synopsis
Target-speaker recognition for multi-talker meetings previously decoded each speaker independently and was confined to 30-second windows. Pandemonium moves the diarization conditioning onto the longform Parakeet backbone, which decodes a whole conversation in one pass, and adds cross-speaker attention plus a mask refiner so that the speakers of one conversation become visible to each other inside the encoder. Trained on synthetic, pseudo-labeled and real data, it reaches state-of-the-art results across several meeting benchmarks, with an average cpWER of 19.2 under predicted diarization.
Interpretation
Multi-talker transcription moves from running one pass per speaker with no information exchanged between passes to letting each speaker's representation attend along the speaker dimension inside a shared encoder, while the decoding unit grows from 30-second windows to whole conversations. Earlier DiCoW-style target-speaker systems emitted only the target speaker's transcript, but their forward passes never exchanged information, discarding cues from overlap, turn-taking and back-channels, and both training and decoding were fixed to the 30-second windows inherited from Whisper. With oracle diarization, Pandemonium reaches cpWER of 14.1, 13.4, 13.3, 11.1, 13.8 and 22.4 on NSF, AMI, ICSI, Mx6, SBC and DiPCo, averaging 14.7, below DiCoW v3.3 at 16.3 and SE-DiCoW at 16.2; with predicted diarization it averages 19.2 against 20.0 for DiCoW v3.3.
Cross-speaker interaction is carried by two modules: one applies multi-head attention along the speaker axis at each frame, and the other re-estimates each speaker's activity from those interacted states and rebuilds the conditioning mask. The speaker conditioning is built from four frame-level events supplied by the diarizer, silence, target, non-target and overlap, but an overlap frame gives every overlapping speaker the same mask, so the conditioning alone cannot tell them apart. Under synthetic-only pretraining, adding cross-speaker attention and the mask refiner gives an average cpWER of 13.1, whereas the same architecture trained on 30-second windows falls back to 17.1, showing the gain depends on longform context; training uses the 4.6k hours of pseudo-labeled conversation that remain from 5.4k hours mined from YODAS2.
Perspective
The demonstrated scope is meeting and conversation-style multi-talker recordings: training uses 115 hours of real meeting data, 4k hours of synthetic data and 4.6k hours of pseudo-labeled YODAS2 conversation, and evaluation covers NSF, AMI, ICSI, AISHELL-4, AliMeeting, SBC, Mixer6 and DiPCo, with concatenated minimum-permutation word error rate (cpWER) as the metric and oracle diarization unless stated otherwise. The authors propose fusing DiCoP with a conversation-pretrained language model, recovering TDT training via a pruned transducer loss, and training diarization and recognition jointly as future work.
The authors state that switching to CTC training costs a small accuracy loss relative to TDT training on short chunks, and say this loss is outweighed by the gains from the interaction modules; the DiCoP baseline remains the stronger TDT-trained model, which favors the baseline. Pseudo-labels alone trail synthetic-only data, which the authors attribute to the English-skewed evaluation hiding cross-lingual gains because Parakeet does not support Mandarin. Diarization and recognition are still trained separately, with joint training listed as future work.