Minho Jeong

This work studies how to distill tabular foundation models (TFMs), whose in-context learning makes inference expensive, into lightweight dataset-specific students, and derives a recipe that uses the full labeled training set as teacher context and trains students solely on teacher predictions for observed and synthetic queries; on TabArena the resulting students outperform supervised tuned-and-ensembled counterparts by 57-98 Elo points, and applied unchanged to TALENT the recipe improves matched default students on 236-258 of 300 datasets while reducing median primary error by 4.0-6.4% and achieving median inference speedups of 3.0-21.6 times over their teachers.

This work studies how to distill tabular foundation models (TFMs), whose in-context learning makes inference expensive, into lightweight dataset-specific students, and derives a recipe that uses the full labeled training set as teacher context and trains students solely on teacher predictions for observed and synthetic queries; on TabArena the resulting students outperform supervised tuned-and-ensembled counterparts by 57-98 Elo points, and applied unchanged to TALENT the recipe improves matched default students on 236-258 of 300 datasets while reducing median primary error by 4.0-6.4% and achieving median inference speedups of 3.0-21.6 times over their teachers.