Straighter video representations are not always better: adversarial training straightens CNNs and transformers, but segmentation and frame prediction do not want it
Synopsis
Human vision is thought to use a straightened encoding of the natural visual world as it evolves in time, meaning frame-to-frame changes follow a more linear path in representation space. A study compared network architecture, different types of robustness, biologically inspired filtering and representational straightness, finding that adversarial training yields straighter representations in both CNNs and transformer-based architectures, but the effect is task-dependent and does not extend to segmentation or frame prediction, nor to other kinds of robustness. Straighter representations also give class predictions temporal stability, including on out-of-distribution data.
Interpretation
Representational straightness is treated as a computable measure of a vision network's robustness and stability under time-varying input, not only as a resemblance to human vision. Humans have been shown to use a straightened encoding of the natural visual world as it evolves in time, while many deep convolutional networks proposed as models of human visual processing do not straighten natural videos. On discrete video sequences, straightness means frame-to-frame changes follow a more linear path at progressively deeper levels of processing; adversarial training leads to straighter representations in both CNN and transformer-based architectures.
Straightness is not universally favorable; whether it appears depends on what a task needs to predict. The straightening effect of adversarial training is task-dependent and does not generalize to tasks such as segmentation and frame prediction, where straight representations are not favorable for predictions. The effect also does not extend to other types of robustness, while straighter representations impart temporal stability to class predictions, even for out-of-distribution data.
Biologically inspired elements raise straightness in a network's early stages, but that gain does not automatically reach downstream layers of CNNs. Architecture, robustness type and biologically inspired filtering mechanisms are compared against straightness together rather than one at a time. Biologically inspired elements increase straightness in the early stages of a network but do not guarantee increased straightness in downstream layers of CNNs.
Perspective
The discussion is limited to representational straightness on discrete video sequences and its relationship to network architecture, robustness types and biologically inspired filtering mechanisms; it addresses researchers evaluating visual network representations.
Straightness is presented with both strengths and limitations as a way of evaluating network representations; the straightening effect of adversarial training is task-dependent and does not generalize to segmentation and frame prediction or to other robustness types; biologically inspired elements raise straightness only in early stages and do not guarantee it in downstream layers.