LLM evaluation drops the single score: 60 models and 44 benchmarks collapse into 8 skill dimensions, and equal-scoring models differ

The score matrix of large language model evaluation is intrinsically low-rank, so a small number of shared dimensions explains most cross-task performance and many benchmarks measure overlapping abilities. Evaluation had relied on a growing collection of benchmarks and on aggregate average scores, while what those scores capture about underlying capabilities remained unclear. Factor analysis of a 60-model by 44-task performance matrix yields eight latent dimensions, with the number of factors chosen by Kaiser's λ>1 rule and an 85% cumulative variance threshold.

The eight dimensions are named as interpretable skills: general NLU, entailment and bias, long-document comprehension, instruction following and generation, domain-knowledge QA, social and ethical judgment, precision and fidelity, and grad-level reasoning. The prior practice was to report long lists of task-specific scores or a single aggregate average, leaving the link between tasks and abilities unclear. Each factor is labeled from its highest-loading tasks, with names proposed by GPT-4o and DeepSeek-R1 from summaries of representative tasks and then refined by the authors.

The low-rank skill space directly supports three evaluation tools: judging whether a new benchmark is redundant, profiling a new model from a small task subset, and selecting a model for a new task. The usual route runs a new model across a broad benchmark suite, which is costly and time-consuming. In leave-one-model-out evaluation over 20 held-out models the mean reconstruction error was 0.095 and only bloomz-1b7 exceeded the threshold; in leave-one-task-out evaluation mean Pearson correlation exceeded 0.85 for well-represented tasks.

The analysis rests on a performance matrix of 60 instruction-tuned models and 44 public tasks, and the skill dimensions are extracted within that task suite; the resulting granularity reflects a balance among statistical support, robustness, and interpretability rather than a unique correct skill resolution. All three tools are validated on held-out data: new-model profiling uses 12 high-communality tasks, and model selection for a new task uses 12 models to estimate task loadings. The authors propose item-level analysis as a next step and extension to multilingual and multimodal benchmarks.

The authors state that skill labels are used only for exposition, and that factor extraction, loadings, and model scores do not depend on the specific phrasing; the eight-factor solution shows skill merging or loss under under-extraction and redundant near-orthogonal axes under over-extraction. The comparison with Chatbot Arena covers only the 13 models shared by both settings, where the aggregate skill score reaches a Spearman correlation of 0.73 with Arena Elo, while individual skills differ widely, with entailment and bias correlating negatively. The authors also note that preference-based evaluation favors fluency and apparent confidence and may underweight capabilities that require careful verification.