Adrian Bulat

A vision Transformer's depth-wise stack is compressed into a single recurrent Transformer block, where the FFN at each recurrent depth is a convex combination of a small shared expert bank programmed by a normalized depth coordinate, so shared weights still yield depth-dependent computation. Under supervised ImageNet-1k training, reViT-B/16 reaches DeiT III accuracy with about 70% fewer stored parameters; under distillation that matches only the DINOv2 teacher's final-layer features, an 8-expert model approaches the teacher on frozen classification, segmentation, and depth probes. One checkpoint also runs at multiple tested depths by resampling the depth coordinate.

A vision Transformer's depth-wise stack is compressed into a single recurrent Transformer block, where the FFN at each recurrent depth is a convex combination of a small shared expert bank programmed by a normalized depth coordinate, so shared weights still yield depth-dependent computation. Under supervised ImageNet-1k training, reViT-B/16 reaches DeiT III accuracy with about 70% fewer stored parameters; under distillation that matches only the DINOv2 teacher's final-layer features, an 8-expert model approaches the teacher on frozen classification, segmentation, and depth probes. One checkpoint also runs at multiple tested depths by resampling the depth coordinate.