Speaker verification gets a cross-scenario scoreboard: 26 trial sets, 11 scenario groups, and profiles that reorder the ranking

The unit of speaker verification evaluation expands from a single public protocol to a fixed collection of cross-scenario trial sets, so systems are compared by trial-set metrics, scenario profiles, and macro summaries rather than by one aggregate score. VoxCeleb-style evaluation remains a stable anchor for English in-the-wild media speech, and scenario-specific evaluations and challenges are valuable, but neither directly shows whether the same system stays reliable across diverse deployment scenarios. The fixed core draws on 16 public sources and contains 26 trial sets, 11 scenario groups, and 9,150,366 verification trials, with target/non-target keys, metadata, and checksums frozen; 18 public checkpoints are evaluated on the same trial sets under one scoring protocol.

Results separate generalization gaps at two levels: similar macro EER does not mean the same failure modes, and trial sets from different sources under one scenario label can order systems differently. A macro ranking gives a compact ordering but cannot replace scenario-wise diagnostics, because group averages hide rank reversals between trial sets. Genre-shift and accent/dialect have the highest mean EER at 27.36% and 24.76%; channel/device, short-duration, overlap, and distance form a second high-error cluster with mean EER between 15.49% and 17.58%; aging and channel/device show high ranking agreement across trial sets, while short-duration, in-the-wild, and overlap contain low-correlation pairs.

Profile differences cannot be attributed to architecture or to whether a training source overlaps the benchmark alone, since trade-offs appear even within one model family. Architecture labels and macro scores are both too coarse to explain scenario profiles, and source overlap is visible but not sufficient to explain deployment generalization. CAMP-Vox and IIC-CAMP share the CAMPPlus lineage, yet IIC-CAMP is better on genre-shift and channel/device while CAMP-Vox is better on accent/dialect, noise/reverb, overlap, and aging; Res34, Res152, and Res293 reach 12.05%, 11.23%, and 11.31% macro EER, so depth is not a monotonic explanation; among the 14 checkpoints whose training descriptions include VoxCeleb, the in-the-wild group gives 2.24% on the VoxCeleb side versus 16.09% on the CN-Celeb side, while short-duration means are close at 16.06% and 17.85%.

The leaderboard is built as a diagnostic interface rather than only a ranked list: new systems submit score packages, are evaluated with frozen keys and common scoring, and receive trial-set breakdowns, scenario-group metrics, macro EER/minDCF, model-wise profiles, and scenario-wise comparisons. Public ranking is only one view of the benchmark output; users also need trial-set breakdowns and scenario-level tables to locate failure modes under particular deployment shifts. The submission kit standardizes data preparation, score generation, prechecks, and score package creation, matches scores by utterance-pair keys, and does not redistribute source recordings; the leaderboard evaluates submitted systems on the fixed public core and reports the same trial-set, scenario-group, and scenario macro views used in the evaluation.

The current fixed core targets cross-scenario speaker verification evaluation, for researchers and engineers who need to compare systems across deployment conditions. It retains widely used protocols such as VoxCeleb1-O/E/H, CN-Celeb, and 3D-Speaker views for comparability with prior work, but treats none of them as the full evaluation target. Scenario groups should be read as generalization views rather than isolated causal factors, because public sources can entangle language, channel, genre, distance, noise, reverberation, and speaking style. Later reviewed public contribution packages may broaden coverage without changing the fixed-core results reported here.

Scenario groups are not causal isolations, since public sources can vary language, genre, channel, distance, noise, reverberation, speaking style, and collection protocol at once. Coverage is uneven, and some scenario groups currently rely on a single public source or trial set, so their average difficulty reflects both the intended mismatch and the source protocol. Complete training manifests are not always available for released checkpoints, so source overlap can be documented when known but cannot always be fully controlled. Ranking still depends on scenario composition: over 10,000 bootstrap resamples of the 11 scenario groups, W2V-BERT2 remains first in 84.3% and in the top three in 99.4%. After removing VoxCeleb trial sets, W2V-BERT2, ReDimNet2, and SAM-ResNet100 change from 9.96%, 10.53%, and 10.58% to 10.44%, 11.14%, and 11.08%.