# Tabular Foundation Models

Across 160 classification and 81 regression datasets, tabular foundation model rankings change with the benchmark and the reliability metric, and regression models that already sit near nominal marginal coverage still under-cover in their worst feature slab, so model selection cannot rest on aggregate predictive accuracy alone.

Across 160 classification and 81 regression datasets, tabular foundation model rankings change with the benchmark and the reliability metric, and regression models that already sit near nominal marginal coverage still under-cover in their worst feature slab, so model selection cannot rest on aggregate predictive accuracy alone.

Tabular foundation model pretraining has scored query predictions against sampled hard labels, so a fixed input keeps supervision from only one class and sampling variability stays in pretraining. GIST designs synthetic task generation and query supervision together: on tractable branches it averages over the conditional label distribution using generator states retained during synthesis, and an entropy budget calibrates the noise scale of the category encoding. Under matched pretraining schedules, TabICLv2, TabSwift and TabPFN-2.5 all improve mean rank on TabArena and TALENT without changing architecture or inference, winning 62.1% of paired dataset-level comparisons across these benchmarks and architectures against the corresponding hard-label controls.

Tabular foundation model pretraining has scored query predictions against sampled hard labels, so a fixed input keeps supervision from only one class and sampling variability stays in pretraining. GIST designs synthetic task generation and query supervision together: on tractable branches it averages over the conditional label distribution using generator states retained during synthesis, and an entropy budget calibrates the noise scale of the category encoding. Under matched pretraining schedules, TabICLv2, TabSwift and TabPFN-2.5 all improve mean rank on TabArena and TALENT without changing architecture or inference, winning 62.1% of paired dataset-level comparisons across these benchmarks and architectures against the corresponding hard-label controls.

This work systematically studies how test-time compute affects predictions of pretrained tabular foundation models along three axes—adaptation, aggregation, and context construction: it introduces DiagScale, a diagonal query-key similarity update training only 0.003-0.03% of parameters that matches full fine-tuning gains across three independently pretrained backbones; on a broader pool of 96 configurations, greedy selection reduces error by 2.4% relative to the default predictor while uniform averaging increases error; attention-guided retrieval improves TabPFN-3 on some large tables, whereas the context expansion methods tested yield no consistent improvement.

This work systematically studies how test-time compute affects predictions of pretrained tabular foundation models along three axes—adaptation, aggregation, and context construction: it introduces DiagScale, a diagonal query-key similarity update training only 0.003-0.03% of parameters that matches full fine-tuning gains across three independently pretrained backbones; on a broader pool of 96 configurations, greedy selection reduces error by 2.4% relative to the default predictor while uniform averaging increases error; attention-guided retrieval improves TabPFN-3 on some large tables, whereas the context expansion methods tested yield no consistent improvement.

Tracing per-query predictive changes through TabICLv2, TabPFN-3, and EXAONE-Tabular, the authors find refinement is highly uneven across depth and often concentrated in later layers; they propose retrospective inference and build Retro, which uses Attention Residuals to adaptively reweight contributions from different depths and query-conditioned gated attention to modulate contextual updates element-wise, shifting refinement earlier and more broadly and reaching top-three overall Elo on TabArena, TALENT, and RelArena while lying on the performance-cost Pareto frontier.

Tracing per-query predictive changes through TabICLv2, TabPFN-3, and EXAONE-Tabular, the authors find refinement is highly uneven across depth and often concentrated in later layers; they propose retrospective inference and build Retro, which uses Attention Residuals to adaptively reweight contributions from different depths and query-conditioned gated attention to modulate contextual updates element-wise, shifting refinement earlier and more broadly and reaching top-three overall Elo on TabArena, TALENT, and RelArena while lying on the performance-cost Pareto frontier.

This work studies how to distill tabular foundation models (TFMs), whose in-context learning makes inference expensive, into lightweight dataset-specific students, and derives a recipe that uses the full labeled training set as teacher context and trains students solely on teacher predictions for observed and synthetic queries; on TabArena the resulting students outperform supervised tuned-and-ensembled counterparts by 57-98 Elo points, and applied unchanged to TALENT the recipe improves matched default students on 236-258 of 300 datasets while reducing median primary error by 4.0-6.4% and achieving median inference speedups of 3.0-21.6 times over their teachers.

This work studies how to distill tabular foundation models (TFMs), whose in-context learning makes inference expensive, into lightweight dataset-specific students, and derives a recipe that uses the full labeled training set as teacher context and trains students solely on teacher predictions for observed and synthetic queries; on TabArena the resulting students outperform supervised tuned-and-ensembled counterparts by 57-98 Elo points, and applied unchanged to TALENT the recipe improves matched default students on 236-258 of 300 datasets while reducing median primary error by 4.0-6.4% and achieving median inference speedups of 3.0-21.6 times over their teachers.

The work introduces fluctuation-supervised pretraining (FSP), labeling each synthetic table by its average treatment effect plus its efficient influence-function fluctuation while deployment remains a frozen forward pass; the author proves an endpoint transition along the path T_λ,P=θ(P)+λP_nψ_P, where every fixed λ<1 retains label ambiguity of order (1-λ)²/n whereas full fluctuation makes the Gaussian label observable and reduces optimal finite-stratum causal label-prediction risk to order n^-2; experiments show that across 24 nonlinear continuous-covariate cells at trained context lengths, continuous-row FSP lowers checkpoint-mean macro RMSE by 7.0% versus S-learner and wins all 12 weak-overlap cells, validation-selected Summary FSP deploys 11.

The work introduces fluctuation-supervised pretraining (FSP), labeling each synthetic table by its average treatment effect plus its efficient influence-function fluctuation while deployment remains a frozen forward pass; the author proves an endpoint transition along the path T_λ,P=θ(P)+λP_nψ_P, where every fixed λ<1 retains label ambiguity of order (1-λ)²/n whereas full fluctuation makes the Gaussian label observable and reduces optimal finite-stratum causal label-prediction risk to order n^-2; experiments show that across 24 nonlinear continuous-covariate cells at trained context lengths, continuous-row FSP lowers checkpoint-mean macro RMSE by 7.0% versus S-learner and wins all 12 weak-overlap cells, validation-selected Summary FSP deploys 11.