# TabPFN-3

Tabular foundation models do in-context learning by carrying every training example through each forward pass, so shrinking the context saves compute but costs accuracy. Activation alignment instead uses the full context to teach a data-constrained student: a single linear transformation is fitted on synthetic unlabeled data to pull the student's intermediate activations toward the teacher's. On 38 TabArena classification datasets with TabPFN-3 and TabFM, the aligned student beats the unaligned baseline across context budgets and recovers nearly half the teacher's advantage in low-data regimes.

Tabular foundation models do in-context learning by carrying every training example through each forward pass, so shrinking the context saves compute but costs accuracy. Activation alignment instead uses the full context to teach a data-constrained student: a single linear transformation is fitted on synthetic unlabeled data to pull the student's intermediate activations toward the teacher's. On 38 TabArena classification datasets with TabPFN-3 and TabFM, the aligned student beats the unaligned baseline across context budgets and recovers nearly half the teacher's advantage in low-data regimes.

This work systematically studies how test-time compute affects predictions of pretrained tabular foundation models along three axes—adaptation, aggregation, and context construction: it introduces DiagScale, a diagonal query-key similarity update training only 0.003-0.03% of parameters that matches full fine-tuning gains across three independently pretrained backbones; on a broader pool of 96 configurations, greedy selection reduces error by 2.4% relative to the default predictor while uniform averaging increases error; attention-guided retrieval improves TabPFN-3 on some large tables, whereas the context expansion methods tested yield no consistent improvement.

This work systematically studies how test-time compute affects predictions of pretrained tabular foundation models along three axes—adaptation, aggregation, and context construction: it introduces DiagScale, a diagonal query-key similarity update training only 0.003-0.03% of parameters that matches full fine-tuning gains across three independently pretrained backbones; on a broader pool of 96 configurations, greedy selection reduces error by 2.4% relative to the default predictor while uniform averaging increases error; attention-guided retrieval improves TabPFN-3 on some large tables, whereas the context expansion methods tested yield no consistent improvement.