Tabular foundation model rankings shuffle across benchmarks: on 160 classification and 81 regression datasets, accurate point predictions do not guarantee trustworthy intervals

The relative ordering of tabular foundation models is not stable: a different benchmark or a different reliability metric puts a different model on top. Earlier tabular benchmarks ranked these models mainly by aggregate predictive performance, as TabArena does, and did not characterize the posterior predictive distributions many of them produce. On 160 classification and 81 regression datasets, the same models exchange positions between TabArena and this benchmark; within this benchmark, LimiX-2 has the lowest mean rank for classification accuracy, TabPFNv3.5 for the Brier score and classwise ECE, and Causilo for confidence ECE.

Regression models already produce intervals close to the target coverage on average, but that average hides under-coverage in local regions of the feature space. Marginal coverage counts the average fraction of covered test points and does not separate subgroups with different feature values. At the 90% nominal level, worst-slab coverage is lower than marginal coverage for every configuration; TabPFNv3.5, TabICLv2 and Causilo keep both close to target, while BART reaches near-target worst-slab coverage through marginal over-coverage and substantially wider intervals.

Most score variation comes from the datasets rather than from average differences among models, so meta-features predict whether models will do well in general more readily than which model will do best. Earlier meta-feature work predicted performance gaps between model families on the 51 TabArena datasets and found limited held-out predictability. For accuracy, Brier score, R-squared and normalized CRPS, meta-features predict the average score shift across models, while the model-specific deviation stays near the baseline; average shifts in coverage error and PIT-KS remain hard to predict even with Extra-Trees.

Growing the inference ensemble mainly improves predictive accuracy and proper scores, whereas direct calibration diagnostics and subgroup coverage do not improve reliably with it. Ensembling and conformal post-processing are often treated as general reliability fixes, but their effects on marginal coverage, subgroup coverage and interval width had not been compared systematically across current models. From 8 to 64 members, median accuracy rises for all 11 classification models and the Brier score improves for all 11, but classwise ECE improves for only 5 classification models and PIT-KS for only 3 distributional regression models; neither global nor covariate-adaptive conformal calibration consistently removes worst-slab under-coverage.

The evaluation covers 160 classification and 81 regression datasets, each sampled to at most 10,000 observations with five train-test splits, and includes 11 classification and 9 regression tabular foundation models plus fixed and tuned BART, RealMLP and XGBoost configurations. Worst-slab coverage measures coverage within a selected feature slab and is not equivalent to pointwise conditional coverage. What remains to be seen is whether these conclusions hold beyond the datasets, configurations and reliability criteria evaluated here.

Worst-slab coverage does not guarantee pointwise conditional coverage, so a near-nominal value does not establish conditional calibration. The score decomposition depends on the set of methods compared, and adding the classical baselines increases the model and interaction shares in regression. The meta-feature associations are exploratory and in-sample, estimating neither independent effects nor causality. Among conformal methods, PCP and, less often, CalLCP and RLCP produce more infinite intervals at higher target coverage, so coverage gains can come at the cost of informativeness.