Straighter video representations are not always better: adversarial training straightens CNNs and transformers, but segmentation and frame prediction do not want it
Human vision is thought to use a straightened encoding of the natural visual world as it evolves in time, meaning frame-to-frame changes follow a more linear path in representation space. A study compared network architecture, different types of robustness, biologically inspired filtering and representational straightness, finding that adversarial training yields straighter representations in both CNNs and transformer-based architectures, but the effect is task-dependent and does not extend to segmentation or frame prediction, nor to other kinds of robustness. Straighter representations also give class predictions temporal stability, including on out-of-distribution data.