Tabular foundation models stop learning only sampled hard labels: mean rank improves across three architectures, with 62.1% of paired comparisons won

Pretraining supervision for tabular foundation models no longer relies only on the single sampled hard label, but averages over the conditional label distribution the generator can supply for the same input. The prior practice fitted query predictions to sampled hard labels; that is an unbiased Monte Carlo estimate of the population loss, but for a fixed input a sampled label selects one class's loss and discards the conditional probabilities the generator could provide for the other classes. Under matched pretraining schedules, TabICLv2, TabSwift and TabPFN-2.5 all improve mean rank on TabArena and TALENT, GIST wins on a majority of datasets in all six architecture-benchmark combinations, and the paired dataset-level win rate across these benchmarks and architectures is 62.1%.

How the conditional target is built depends on whether the sampled label enters input generation: if the label draw does not affect the model input, the sampling probabilities give the target directly; if the sampled class influences generated features, those features carry additional label evidence. Evaluating that posterior directly can require inference through the downstream graph, so tractable targets are constructed only on supported branches while the remaining branches keep hard-label supervision. On supported label-dependent branches the sampled class is mapped to a category code perturbed with Gaussian noise, all label-dependent feature computations route through that encoding, and its local likelihood yields the posterior target; an entropy budget bounds average residual label uncertainty from row probabilities and pairwise code distances, and infeasible budgets fall back to the original encoding and sampled hard labels.

The report covers supervision design during synthetic-task pretraining of tabular foundation models, with evaluation limited to IID classification on TabArena and TALENT and to the temporal and grouped OOD splits of BeyondArena. The method leaves model architecture and inference unchanged, so the conclusions apply to tabular foundation models that use PFN-style synthetic priors. Transfer under distribution shift remains an open direction.

Conditional targets are tractable only on supported label-dependent branches, and the remaining branches keep hard-label supervision, so the average effect depends on how large a share of tasks those branches cover. OOD results vary by architecture and metric: on the 12 datasets shared by all three architectures multiclass log loss falls for all three, while binary AUC improves for TabICLv2 and TabSwift but decreases for TabPFN-2.5. The preferred entropy budget varies by benchmark and metric.