Most LLM benchmark tables are 90% empty: explicit same-capability calibrations fill 71% of missing scores without running evals
Synopsis
In a matrix of 401 models and 146 benchmark versions, only 8.3% of cells are measured; benchgap runs no evaluations and instead fits monotone curves between each pair of same-capability benchmarks on the models measured on both, selecting by leave-one-out error and attaching an error and confidence level to every estimate. In an end-to-end holdout masking 15% of measured scores, it fills 71% of the masked cells at 4.6 percentage points mean absolute error, 23% lower than a rank-2 matrix-completion baseline on the same cells; on the cells it leaves empty, the baseline errs at 11.8 points.
Interpretation
Public leaderboard score matrices are mostly empty, and benchgap treats each empty cell as an estimable quantity filled from already measured scores rather than from another evaluation run. Across the compiled 401 models and 146 benchmark versions only 8.3% of cells are measured, and the median benchmark version has scores for just 16.5 models, so capability profiling and model selection that need a complete matrix start from missing data. In an end-to-end holdout masking 15% of measured cells, the pipeline fills 71% of the masked cells at 4.6 percentage points mean absolute error, 23% below the 5.9 points of a rank-2 matrix-completion baseline on the same cells.
Every estimate traces back to a named source-to-target calibration curve rather than to a single global latent-space model. Matrix completion and scaling-law approaches fit one global model in a latent space and report one aggregate error, whereas each calibration here holds only between one pair of same-capability benchmarks and carries its own leave-one-out error. The snapshot stores 1,147 univariate calibrations with median R² of 0.71 and median leave-one-out error of 5.1 percentage points, plus 21 multivariate mappings with median leave-one-out error of 1.9 points that carry only 1.4% of estimates.
Estimates are published in confidence tiers, and higher tiers show lower realized error. In the holdout, high, medium and low tiers err at 3.1, 5.1 and 7.5 percentage points mean absolute error, so ordering by claimed error matches ordering by realized error. In the same test the realized root-mean-square error is 1.4 to 1.7 times the claimed leave-one-out value, and the high tier's realized 5.1 points exceeds the 5-point threshold that defines it.
The same benchmark disagrees with itself across evaluation harnesses, and the gap is largest on agentic benchmarks. On verified same-item-set version pairs, agentic benchmarks differ by 8.7 percentage points on average against 2.3 points for tool-free ones. On Humanity's Last Exam, enabling tools raises the scores of all 11 models measured both ways, by 10.0 percentage points on average, sign test p = 10⁻³.
Perspective
The demonstrated setting is one data snapshot: 401 models, 146 benchmark versions and 4,887 measured scores, with all numbers referring to data retrieved on 8 October 2026. The intended readers are researchers and engineers comparing models across leaderboards or selecting models, and each estimate refers to a specific benchmark version, predicting what that source's harness would have measured.
Leave-one-out errors are selected minima over curve families and mappings and are therefore optimistic; in the holdout realized root-mean-square error is 1.4 to 1.7 times the claimed value, with heavy tails. Holdout cells come from the same distribution as the fitting data and from its easier end, and reweighting to the published confidence mix gives about 5.9 percentage points mean absolute error for a published estimate. For sparsely measured models with one or two scores, the pipeline fills only 19% to 26% of remaining cells. Multivariate mappings did not reproduce their lower in-sample error out of sample and are reported as unproven.