# Activation Alignment

Tabular foundation models do in-context learning by carrying every training example through each forward pass, so shrinking the context saves compute but costs accuracy. Activation alignment instead uses the full context to teach a data-constrained student: a single linear transformation is fitted on synthetic unlabeled data to pull the student's intermediate activations toward the teacher's. On 38 TabArena classification datasets with TabPFN-3 and TabFM, the aligned student beats the unaligned baseline across context budgets and recovers nearly half the teacher's advantage in low-data regimes.

Tabular foundation models do in-context learning by carrying every training example through each forward pass, so shrinking the context saves compute but costs accuracy. Activation alignment instead uses the full context to teach a data-constrained student: a single linear transformation is fitted on synthetic unlabeled data to pull the student's intermediate activations toward the teacher's. On 38 TabArena classification datasets with TabPFN-3 and TabFM, the aligned student beats the unaligned baseline across context budgets and recovers nearly half the teacher's advantage in low-data regimes.