PrivTab embeds a differential-privacy mechanism inside a tabular foundation model, beating DP-LR and DP-MLP on 33 TabArena datasets under moderate-to-strong privacy while cutting fitting time from minutes to about 10 milliseconds

PrivTab places the privacy mechanism where sensitive information enters the model, using differentially private multi-head cross-attention (DP-MHCA) to compress a sensitive context dataset into a fixed-size private summary whose sensitivity does not grow with context size. Existing tabular foundation models (TabPFN, TabICL) still require sensitive context rows or non-private representations at inference and lack formal privacy guarantees, while traditional private learning relies on dataset-specific DP stochastic optimisation. PrivTab instead produces a reusable private summary once, inside the architecture. The paper provides a per-layer sensitivity theorem (Theorem S3.1) and an end-to-end -GDP guarantee via adaptive composition (Theorem S3.2), formally verifies the sensitivity bound and GDP composition in Lean 4, and adds implementation checks and empirical audits.

Across 33 TabArena classification datasets, six privacy regimes and ten 80/20 context-target splits per dataset, PrivTab achieves higher accuracy and better log loss in the moderate-to-strong privacy regimes and the highest aggregate Elo rating. DP-MLP closes the gap only under weak privacy constraints; PrivTab's advantage is most pronounced in log loss under heavy noise, consistent with training through an embedded privacy mechanism helping preserve reliable uncertainty estimates when privacy bounds are tightest. Comparisons are against carefully tuned DP-LR and DP-MLP trained with DP stochastic optimisation for each dataset and privacy level, whereas PrivTab requires no dataset-specific optimisation.

PrivTab reduces median fitting time to about 10 milliseconds per split, versus roughly 400 and 130 GPU hours for DP-LR and DP-MLP across the benchmark, while PrivTab fitting required only 30 GPU seconds. Traditional methods require repeated dataset-specific training and private model selection across 198 dataset-privacy configurations (1,980 final training runs per baseline), whereas a single pretrained PrivTab variant can be reused across privacy levels without retraining. Runtimes were measured on one Nvidia V100 with identical software and hardware settings, including data transfer and computation; baseline timings include private learning-rate selection and training.

In a likelihood-ratio membership-inference attack on Maternal Health Risk clinical records, conservative lower confidence estimates of fitted attack accuracy approach 100% for the most vulnerable records under TabPFN v3 and TabICL v2, while PrivTab remains much closer to perfect privacy across all evaluated privacy levels. The audit reports record-level vulnerability rather than aggregate averages that can hide highly vulnerable records, and the paper notes that empirical attacks cannot replace formal proofs. The analysis used all 1,014 rows with 4,000 balanced context draws of 507 rows each, identical membership masks for every method, and finite-population correction with delta-method confidence intervals.

The work targets domain practitioners who need classification on sensitive tabular data but are rarely privacy specialists, and downstream users who may rely on standard CPU-based hardware. Its design means that once the sensitive dataset is converted into a private summary, all later predictions are post-processing and incur no additional privacy loss, and the summary can be stored and shared for later predictions; the same pretrained variant can be used across privacy levels without retraining. The private-preprocessing experiments show that, for datasets with public column descriptions detailed enough to infer meaningful clipping intervals and with modest feature counts, an end-to-end private pipeline is feasible.

The study covers only classification and not regression, and PrivTab's privacy model assumes one user per row, so extending the guarantee to settings where one individual contributes multiple rows is less straightforward than for approaches based on DP stochastic optimisation. PrivTab has a smaller relative advantage when privacy is weak or context sizes are large, since models trained separately on each dataset can benefit more from both the relaxed privacy constraint and additional data. The private-preprocessing experiments demonstrate one possible approach, but the appropriate preprocessing pipeline remains application-dependent, and the same preprocessing requirement also applies to the private-learning baselines. Empirical attacks and formal verification cannot fully substitute for validating the running system, and the correspondence between the formal objects and the executable implementation remains subject to implementation review.