Tabular foundation models stop learning only from sampled labels: GIST supervises with generator states and wins 62.1% of paired comparisons across three architectures
Lead
During pretraining of tabular foundation models, the sampling noise of query labels is averaged out using intermediate generator states: on TabArena and TALENT, GIST improves mean rank for all three architectures and wins 62.1% of paired dataset-level comparisons against the corresponding hard-label controls.
Story
Pretraining supervision for tabular foundation models no longer relies only on sampled hard labels; it averages the query loss over conditional label distributions using intermediate states the generator retains while synthesizing a task. The prior practice scored query predictions against a sampled hard label, so even when the generator could supply conditional probabilities for all classes, one sampled label selected a single class's loss. Under matched pretraining schedules, GIST improves mean rank for TabICLv2, TabSwift, and TabPFN-2.5 on both TabArena and TALENT, and wins 62.1% of paired dataset-level comparisons against the corresponding hard-label controls.
Conditional targets are enabled only on branches where the generator yields tractable conditional probabilities, and the remaining branches keep sampled hard labels, with architecture and inference unchanged. When the sampled class does not affect the model input, the sampling probabilities give the target directly; when it does affect generated features, GIST perturbs the category code with Gaussian noise, routes all label-dependent feature computations through that encoding, and reads the posterior target from its local likelihood. The noise scale is calibrated by an entropy budget that bounds average residual label uncertainty in the encoding channel; on TabSwift all three tested budgets beat the hard-label control, with a default budget of 0.15.
What to watch
The work targets tabular foundation models pretrained on synthetic priors, evaluated on IID classification in TabArena and TALENT and on temporal and grouped OOD splits in BeyondArena, across three architectures. Conditional targets are enabled only on branches where the generator yields tractable conditional probabilities, and other branches keep sampled hard labels.
The reported OOD results vary across architectures and metrics: on the 12 datasets shared by all methods, multiclass log loss falls for all three, while mean binary AUC decreases for TabPFN-2.5. The authors list transfer under distribution shift as an open direction.