DiT features lag U-Net because noise rides the main path: long skip connections bypass it and lift mid-layer quality

When a diffusion transformer (DiT) is used as a vision feature extractor, its intermediate activations are weaker than those of a U-Net backbone, and long skip connections are a general contributor to that gap. Earlier work reported weak DiT features, but its explanation applied mainly to semantic correspondence, while the degradation also appears on other tasks, so the cause was not complete. On pretrained models with roughly aligned generation capability, U-ViT with long skip connections reaches 53.53 on SPair (a), while DiT-XL/2 without them reaches 44.01.

Long skip connections let the diffusion noise bypass the middle section that reconstructs the target representation, so activations there no longer have to carry noise as well, and feature quality rises. The common explanation is that long skip connections only shortcut high-frequency semantics such as edges, but the tested tasks are fine-grained, including semantic segmentation, which that explanation cannot cover. Two DiT-XL/2 models were trained from scratch on ImageNet 256×256 with the only difference being the presence of long skip connections; the version with them scores 41.10 on SPair (a) and 36.32 on Horse-21 versus 34.68 and 30.35 without, and a 100k-step version with FID aligned at 21.67 still scores 38.85 and 35.24.

Mutual-information probing and input-substitution controls together support the noise-bypassing mechanism and indicate that long skip connections carry both high-frequency semantics and diffusion noise. Previous knowledge of long skip connections covered high-frequency semantics only, not diffusion noise as information they carry. A mutual information neural estimator measured each activation against the target latent and against the added noise, showing less noise information and more target information in the middle section with long skip connections; replacing the original noise with random noise or the latent with one from another image lowers training loss yet degrades representation and generation, with one setting collapsing to NaN.

The conclusions come from a controlled DiT-XL/2 comparison trained from scratch on ImageNet 256×256, plus comparisons of pretrained DiT-XL/2, U-ViT and Skip-DiT, with downstream evaluation on SPair-71k semantic correspondence, Horse-21 and CityScapes semantic segmentation, and NYU Depth v2 depth estimation. The intended readers are researchers and engineers working on diffusion backbone design or diffusion feature extraction. Attempts to add long skip connections to a pretrained DiT produced only slight feature gains, and the authors consider effective fusion likely to require training from the start.

The authors do not claim long skip connections are the only cause of DiT feature degradation, nor that diffusion noise matters more than high-frequency semantics, only that both are non-negligible. The controlled experiments use the conventional DiT architecture rather than MMDiT, and effectively augmenting existing pretrained DiTs with long skip connections was not achieved.