Smart glasses two-person transcription gets a frontend: separating speakers before the audio-language model cuts errors on overlapped speech

Four-channel audio captured by smart glasses in two-person conversations can be passed through a speech frontend before a large audio-language model transcribes it, yielding time-stamped, speaker-attributed transcripts under overlapping speech, background interference and far-field conditions. Most prior multimodal large-model transcription systems do not explicitly model or separate overlapped speech, relying instead on the backend model to transcribe overlapping regions directly, and many are designed for single-channel input rather than the spatial information from multiple glasses-mounted microphones. On the official validation set, both frontend-assisted systems beat the fine-tuned system without a frontend: the guided source separation system reaches a tcpCER of 6.97 with estimated voice activity labels and the multi-channel separation system 7.33, against 7.82 for the fine-tuned SoulX-Transcriber.

The frontend splits one mixed recording into two single-speaker tracks, which are decoded separately and then merged into the complete dialogue. Guided source separation uses pre-computed speaker activity information to guide blind source separation and beamforming, while the multi-channel separation network uses dual-path recurrent blocks to estimate two complex-valued time-frequency masks, one per speaker. The validation set is divided by overlap ratio into low, mid and high groups of 26, 25 and 25 chats with overlap ratios of 0.1%–2.9%, 3.0%–11.9% and 12.0%–22.8%, and the advantage of frontend processing is more evident on the higher-overlap chats.

The final submission uses the fine-tuned SoulX-Transcriber rather than a frontend system, because it is the most stable on full-length audio. Audio processed by the frontend tends to cause more hallucinations when fed directly to the pretrained model, showing up as repeated token generation or obvious timestamp drift, and fine-tuning plus decoding adjustments can reduce them. That final system achieves a tcpCER of 6.22 on the test set.

The report targets the time-stamped speaker-attributed transcription task for two-person smart-glasses conversations, with validation and test both run under the challenge data and metrics. The frontend systems are evaluated on the validation set by overlap-ratio group, where the advantage concentrates on high-overlap chats; the final submission is the fine-tuned system chosen for stability on full-length audio.

SoulX-Transcriber occasionally hallucinates, showing up as repeated token generation or obvious timestamp drift; distortion from frontend processing and zero-padded non-speech regions shift the processed audio away from the pretraining distribution, causing more hallucinations when fed directly to the pretrained model. A few problematic samples were handled by inference on roughly 30-second segments, and with oracle voice activity labels the DER can still be non-zero because timestamps are generated autoregressively by the backend.