Patrik P. Süli

In a matrix of 401 models and 146 benchmark versions, only 8.3% of cells are measured; benchgap runs no evaluations and instead fits monotone curves between each pair of same-capability benchmarks on the models measured on both, selecting by leave-one-out error and attaching an error and confidence level to every estimate. In an end-to-end holdout masking 15% of measured scores, it fills 71% of the masked cells at 4.6 percentage points mean absolute error, 23% lower than a rank-2 matrix-completion baseline on the same cells; on the cells it leaves empty, the baseline errs at 11.8 points.

In a matrix of 401 models and 146 benchmark versions, only 8.3% of cells are measured; benchgap runs no evaluations and instead fits monotone curves between each pair of same-capability benchmarks on the models measured on both, selecting by leave-one-out error and attaching an error and confidence level to every estimate. In an end-to-end holdout masking 15% of measured scores, it fills 71% of the masked cells at 4.6 percentage points mean absolute error, 23% lower than a rank-2 matrix-completion baseline on the same cells; on the cells it leaves empty, the baseline errs at 11.8 points.