Tabular foundation models can drop 90% of context: one linear alignment layer, 10% of data matches two to four times more
Synopsis
Tabular foundation models do in-context learning by carrying every training example through each forward pass, so shrinking the context saves compute but costs accuracy. Activation alignment instead uses the full context to teach a data-constrained student: a single linear transformation is fitted on synthetic unlabeled data to pull the student's intermediate activations toward the teacher's. On 38 TabArena classification datasets with TabPFN-3 and TabFM, the aligned student beats the unaligned baseline across context budgets and recovers nearly half the teacher's advantage in low-data regimes.
Interpretation
Prediction cost in tabular foundation models grows with the number of context examples, because every training example passes through the forward computation at each prediction; activation alignment lets a model behave as if it had seen the full context while conditioning on only a subset. Earlier routes compress the context or prune redundant transformer layers, and both demand substantial GPU training; activation alignment changes neither the input nor the architecture, only internal representations. On 38 TabArena classification datasets with TabPFN-3 and TabFM, the aligned student wins most comparisons against the unaligned baseline across all evaluated context budgets, with reported statistical significance.
The aligner is a single linear map that corrects the intermediate activations of a context-limited student toward those of a full-context teacher. Training uses only synthetic unlabeled query vectors, needs no extra labels, and does not backpropagate through the model. The aligner is fitted by mean squared error on pre-extracted activations and converges in seconds to minutes on commodity CPU cores; evaluation spans 38 datasets and five context fractions, counting only cases where the teacher beats the baseline.
In low-data regimes, an aligned student with 10% of the context reaches the predictive level of an unaligned student given two to four times as many examples. The unaligned student degrades substantially as context shrinks, while the aligned student keeps its median effective sample fraction above parity across all evaluated budgets. At 10% context, median effective sample fractions are about 3.2 for TabPFN and 3.3 for TabFM; the aligned student also beats XGBoost trained on the full data in over 80% of evaluated conditions.
Perspective
Evaluation is limited to classification, covering 38 TabArena datasets with the two tabular foundation models TabPFN-3 and TabFM, with context fractions mainly from 10% to 50% and full-spectrum results reported. The aligner needs offline access to the full dataset to extract teacher activations, so it addresses inference-time compute and memory bottlenecks rather than genuinely scarce data. The authors propose composing the aligner with context compression, layer pruning, or retrieval-based context selection, and exploring non-linear aligners, multi-layer intervention, and end-to-end distillation objectives.
The authors state the method has not been verified on regression and has not been validated across a broader family of tabular in-context learning architectures. Five of the 38 datasets show negative mean effective-sample gain for both models, and no characterization yet separates them in advance. The aligner requires the full dataset to extract teacher activations, so it does not apply when only a small dataset exists to begin with.