TT-VidT splits appearance and motion into two pathways: among 24 architecture-objective pairings, only TT3D with Diff Compression leads Jester, SSv2, ARID and Diving48 at once

The paper introduces a controlled video self-supervised learning protocol that holds architecture, objective, data exposure, schedule and parameter scale fixed under one recipe, comparing 4 encoders by 6 objectives across 24 cells, supplemented by a single-frame DINOv3 appearance diagnostic to contextualize boundary cases. Prior video SSL comparisons often vary architecture, objective, data exposure, schedule and decoder capacity at once, making it hard to attribute motion-prioritized behavior to a specific architecture-objective choice; this work compresses the variables onto the architecture and objective axes. The sweep runs at roughly 170M-190M encoder scale on 1.7M clips for 8 epochs, about 13.6M samples and about 425k optimizer steps under the shared recipe, with a single run per cell and multi-seed checks on the canonical cells.

TT-VidT combines TT3D and Diff Compression: TT3D adds a compact Temporal Transfer Layer on top of a DINOv3-initialized ViT-B/16 spatial path, running a block-causal 3D self-attention over downsampled per-frame spatial tokens together with motion tokens; Diff Compression reconstructs target frames from the first frame's spatial features as appearance anchor and the t-th frame's motion embedding as the carrier of frame-specific information. Unlike VTok, which computes the motion token by an explicit feature subtraction followed by a projection, here the motion token is the direct output of the Temporal Transfer pathway learned end-to-end under the reconstruction objective, with no built-in feature diff; the decoder still cross-attends to the full spatial features, so the appearance pathway stays wide while the motion pathway stays compact. The method is instantiated in TT1D and TT3D forms, with a temporal-attention sequence of 192 tokens, well below the length that attention over full spatial features would require; encoder, Temporal Transfer Layer, decoder and motion-token embeddings are trained jointly.

The sweep shows the gain comes from the pairing of TT3D and Diff Compression rather than either component alone: with TT3D fixed, replacing Diff Compression by MAE drops Jester from 53.89 to 11.07 and SSv2 from 18.28 to 4.98; with Diff Compression fixed, replacing TT3D by ViT3D or DisMo gives 19.82 or 12.98 on Jester and 6.57 or 5.53 on SSv2. Only ViT3D+MAE and TT3D+Diff Compression reach a high tier on the motion-focused Jester and SSv2 benchmarks in the no-augmentation condition, so the successful region of the 24-cell grid is sparse and the pairing shows an interaction effect. The conclusion rests on the no-augmentation sweep of Table 1 with a common decS_imgnet decoder and a single run per cell; on ARID the picture is more mixed, with ViT3D+MAE leading at 24.37 against 21.98 for TT3D+Diff Compression.

In the final comparison TT-VidT is the only method to lead Jester, Something-Something V2, ARID and Diving48 fine-tuning simultaneously, reaching 73.25 on Jester and 25.92 on SSv2 under frozen attentive probing and staying 21.0 and 12.9 points ahead of the strongest baseline after end-to-end finetuning; a motion-inversion probe shows it keeps its original answer on at most 1% of flipped or time-reversed SSv2 mirror-class clips while every baseline keeps it on 9%-43%. The lead is not only in average accuracy but in whether the representation itself reads motion direction: the single-frame DINOv3 control scores 0.0 flip and 100.0 stay under reversal, validating the probe and showing that Temporal Transfer and Diff Compression turn an order-blind substrate into the most direction-faithful encoder of the comparison. The final comparison covers frozen attentive probing on HMDB, ARID, IARD, Jester, SSv2 and EK-V verb classification plus Diving48 full fine-tuning and EK-V anticipation; the Jester and SSv2 lead holds across all six probes (kNN, linear, MLP, layer-weighted variants and attentive), and the weakest of nine TT-VidT measurements stays above the strongest ViT3D+MAE measurement.

The result is aimed at researchers and engineering teams studying video self-supervised representations under a matched small recipe: within roughly 170M-190M encoder scale, 1.7M OpenVid and Moments-in-Time v2 clips and 8 epochs, decoupling an appearance anchor from a compact motion pathway yields both motion sensitivity and efficiency, with TT3D costing 456.1 GF encoder FLOPs, about half of DisMo and less than half of VideoMAE. For downstream tasks that need motion cues, such as robotics, assistive perception, sports or skill analysis and scientific video understanding, this offers a cheaper operating point; for tasks where identity, object and scene cues matter more, the paper explicitly names HMDB51, IARD and EPIC-Kitchens as boundaries, where VideoMAE leads HMDB51 at 27.73, DisMo leads IARD at 89.89 and V-JEPA 2 leads EK-V anticipation at 23.07. The authors also frame concrete next steps: combining TT3D with DisMo-style dual augmentation, testing longer pretraining and stronger data mixtures, and redoing the diagnostic with alternative appearance references such as DINOv2 or CLIP.

Several open questions remain for a careful reader. The study itself is scoped to one matched recipe and one roughly 170M-190M encoder scale, with mostly single-run sweep cells, so its profile should be read as evidence about this controlled regime rather than a claim about all data scales or schedules. The appearance-vs-motion diagnostic is described by the authors as a lightweight observation using a single-frame DINOv3 attentive probe as the appearance reference, so alternative references such as DINOv2 or CLIP may shift dataset positions; Diving48 sits below the diagonal in the frozen-probe view and its gain comes from full fine-tuning, so it is treated as complementary evidence rather than a diagnostic point. IARD variance comes mostly from which of the five actors is held out, and under one fixed actor split all models land at a similar level. Combinations such as TT3D with DisMo-style dual augmentation, partial unfreezing of the spatial encoder, richer temporal readouts, and evaluation on untrimmed or egocentric video are untested. This evidence bundle is the full text, but per-item numbers in tables and appendices are cited as reported in the main text; exact reproduction should return to the original tables.