University of Technology Nuremberg

Repurposing pretrained tabular foundation models as encoders of entire supervised datasets maps an unseen dataset to a fixed-dimensional vector in a single forward pass, reaching the strongest aggregate rank across six dataset-level tasks.

Repurposing pretrained tabular foundation models as encoders of entire supervised datasets maps an unseen dataset to a fixed-dimensional vector in a single forward pass, reaching the strongest aggregate rank across six dataset-level tasks.

The authors build WT++, a 95-hour urban walking-tour video dataset, benchmark MoCo v3, DINO and MAE under a strict sliding-window streaming regime without global shuffling or multi-epoch replay, find that contrastive and self-distillation methods lag while MAE is more robust but still falls short of i.i.d. pretraining, use a controlled experiment to attribute the gap to high intra-batch rather than inter-batch similarity, and propose StreamMAE, which keeps the MAE reconstruction objective while adding stream-aware regularization and motion-biased crop selection, outperforming streaming baselines, matching same-data i.i.d. MAE, and improving as the pretraining stream grows from 12 to 95 hours.

The authors build WT++, a 95-hour urban walking-tour video dataset, benchmark MoCo v3, DINO and MAE under a strict sliding-window streaming regime without global shuffling or multi-epoch replay, find that contrastive and self-distillation methods lag while MAE is more robust but still falls short of i.i.d. pretraining, use a controlled experiment to attribute the gap to high intra-batch rather than inter-batch similarity, and propose StreamMAE, which keeps the MAE reconstruction objective while adding stream-aware regularization and motion-biased crop selection, outperforming streaming baselines, matching same-data i.i.d. MAE, and improving as the pretraining stream grows from 12 to 95 hours.