Finnish speech models stop picking only clean broadcasts: 471K hours kept from 2.1M hours of raw audio by recognition confidence
Synopsis
Finnish speech self-supervised pretraining no longer relies only on voice activity detection and language identification to pick data. Those pipelines ensured speech was present, not that it was clear; this pipeline uses ASR confidence as a proxy for pretraining data quality after voice activity detection and deduplication, keeping 471K hours from 2.1M hours of raw broadcast audio. On four benchmarks, confidence filtering gives 3-7% relative WER improvement over language-identification filtering despite a smaller retained training set.
Interpretation
Broadcast archives can serve as a usable source for speech self-supervised pretraining when filtering is strict enough. Prior monolingual SSL studies either used all available data to maximize scale or relied on selected broadcast programs and curated datasets, typically with voice activity detection or Whisper-based utterance extraction, which target speech extraction rather than systematic selection by speech quality. Training wav2vec 2.0 Base Transformer models on deduplicated VAD data, confidence-threshold filtering at similar retention (~72%) gives lower WER than LID-only filtering on three of four benchmarks.
The filtering pipeline has three stages: cut out speech, remove repeated content, then keep clear speech by recognition confidence. Voice activity detection with pyannote.audio yields 1.2M hours of speech segments split into chunks of at most 30 seconds; jingles, advertisements and rebroadcast programs create heavy repetition, so transcript-based deduplication using Finnish wav2vec 2.0 Large transcripts keeps one instance per unique transcript, cutting 1.2M hours to 890K hours. Confidence is the mean log-probability of the predicted frame-wise character sequence excluding blank tokens; transcript analysis showed some phrases repeat tens of thousands of times across channels and time periods, and LID analysis of the final 471K hours shows 99% Finnish content.
At matched parameter budgets, convolution-augmented encoders clearly outperform the standard Transformer at Base scale. Prior SSL studies varied architecture and model size together, for example Conformer-based and Transformer-based wav2vec 2.0 variants using 618M and 317M parameters respectively, making it unclear whether gains stem from architecture or capacity. Transformer, Macaron-Net, Conformer and E-Branchformer are compared within a unified SSL framework at matched parameter counts from 95M to 965M, with convolution-augmented encoders reducing WER by up to 37% relative over Transformer at Base scale.
Perspective
The pipeline and conclusions rest on the broadcast archive of the Finnish National Audiovisual Institute, covering 26 TV channels, 85 radio stations and online media content from 2009 to 2025. Evaluation spans Finnish Parliament, Lahjoita puhetta, Common Voice 16.1, FLEURS and a manually transcribed KAVI test set of two one-hour radio broadcasts. The broadcast archives cannot be redistributed due to copyright and broadcast licensing restrictions, but the models are released publicly. The authors state that while the study focuses on Finnish, the findings offer guidance for SSL research on other languages with unconstrained audio sources.
The authors note that transcript-based deduplication may reduce acoustic diversity for repeated phrases spoken by different speakers, but they prioritize removing extreme redundancy. At Large scale, 36-layer variants achieve competitive in-domain performance but generalize inconsistently: Conformer degrades 5.8% relative on KAVI and E-Branchformer underperforms 21% relative on Common Voice. At X-Large scale, convolution-augmented encoders scale less uniformly than Transformer, with domain-specific degradations.