Dropping a "redundant" benchmark can delete the wrong one: two tests correlated at 0.86 barely agree on top models
Synopsis
Pruning a benchmark because it correlates with another may discard exactly the signal that separates frontier models. Across 4,497 models on Open LLM Leaderboard v2, BBH and GPQA correlate at 0.861 overall, yet conditional Spearman falls to 0.434 among the top 20% and to -0.278 among the top 5%; 7 of 15 pairs are best fit by zero-tail-dependence Frank copulas, so high bulk correlation does not imply agreement at the frontier.
Interpretation
High overall correlation between two benchmarks does not guarantee that top models rank consistently across them. Standard practice treats a pair with Spearman above 0.8 as redundant and drops one benchmark, assuming dependence is uniform across the performance spectrum. On 4,497 models from Open LLM Leaderboard v2, BBH and GPQA correlate at 0.861 overall, while conditional Spearman falls to 0.434 in the top 20%, -0.04 in the top 10% and -0.278 in the top 5%.
The attenuation comes from how dependence is structured in the tails, not from the size of the bulk correlation alone. A copula separates each benchmark's score distribution from the dependence between the two, and a tail dependence coefficient describes how likely a model at the top of one benchmark is also at the top of the other. All 15 pairs are better fit by non-Gaussian copulas, and 7 of 15 (47%) are best fit by Frank copulas with zero asymptotic tail dependence, with ΔBIC of 237 for BBH and GPQA and 585 for MATH and MMLU-Pro.
The ranking collapse survives removing fine-tuned variants that share a base model. The original 4,497 models include many fine-tunes of the same base model, which overstates effective degrees of freedom. After deduplication to 779 base-model families, cluster bootstrap family stability is 98% for BBH and MUSR and 96% for MATH and MMLU-Pro, with ΔBIC intervals entirely above zero; BBH and GPQA is weaker at 76% with an interval spanning zero.
Perspective
The findings rest on the six reasoning and knowledge benchmarks of Open LLM Leaderboard v2, with directional replication on BenchPress (316 benchmarks) and kensho/WILD (27 benchmarks, 65 models from independent organizations); multimodal or generation benchmarks are not covered.
A tail dependence coefficient of zero is a parametric property of the Frank family rather than an independent measurement; nonparametric 5% empirical tail concentration ranges from 0.06 to 0.61, so tail dependence is sharply reduced at finite sample size rather than literally zero. Whether the frontier ranking collapse reflects capability specialization or training-data composition remains an open question in the source.