How much of a conv layer's own input survives its output: compression beats depth as a predictor, and copying pretrained singular vectors raises recoverability
Synopsis
Across nine ImageNet convolutional networks, how much of a layer's own input patches can be reconstructed from its output falls with depth but is more consistently predicted by the layer's effective compression, and copying pretrained singular vectors onto a random encoder raises recoverability while copying the pretrained singular-value spectrum or activation sparsity lowers it.
Interpretation
How much of the image content a convolutional layer reads survives in its output can be quantified layer by layer: encode the input patches that layer reads with the layer's weights, normalization and activation, fit a decoder to reconstruct those patches, and report the coefficient of determination on held-out patches as recoverability. Earlier information analyses of deep networks mostly measured mutual information between hidden representations and the whole input image or the labels, which is hard to estimate for high-dimensional deep feature maps and depends heavily on the estimator, and early analyses were limited to small fully-connected networks. Probing every convolutional layer of nine ImageNet-pretrained CNNs, recoverability falls from about 1.0 in the first layer to between 0.1 and 0.4 in the last layer for VGG16, ResNet18, ResNet50, GoogLeNet, SqueezeNet, AlexNet and DenseNet121, decreases less in MobileNetV2 (0.99 to 0.66), and rises in EfficientNet-B0's last layer to 0.79 from the previous layer's 0.49.
A layer's recoverability is set mainly by how much it actually compresses its input, not by its depth. A layer's type is set by channels, kernel size and stride, which also fix a nominal compression ratio, but input patches may occupy far fewer directions than that nominal dimension, so nominal compression can mislead. Defining effective compression as the number of principal components explaining 98% of the variance in a layer's input patches relative to its width, recoverability falls as effective compression rises in every architecture, with negative Spearman correlation across models, and effective compression predicts recoverability better than nominal compression in all models; four ResNet50 stage-2 layers with identical patch dimension, width and nominal compression have recoverability 0.52, 0.85, 0.52 and 0.38, matching a dip in effective compression.
The recoverability gained from training is not uniform, and transferring a single property of the pretrained weights onto a random encoder moves recoverability in a consistent direction. The difference between a pretrained encoder and a same-shape random encoder, called learning-attributed recoverability, is negative in 10 of 13 models and ranges from negative in AlexNet to positive in ResNet18 trained on CIFAR-10, so the gap alone says little about what training changed. Matching the pretrained activation sparsity lowers recoverability in all 13 models, matching the pretrained singular-value spectrum also lowers it, matching the pretrained BatchNorm parameters lowers it in 9 of 10 models with BatchNorm, and copying the pretrained singular vectors raises recoverability in 11 of 13 models.
Perspective
The work measures the recoverability of a convolutional layer's own input patches and applies to layer-wise analysis of convolutional network representations, for readers who study internal representations and the information bottleneck. The probe scales with channels and kernel size, so it reaches deep CNNs with wide feature maps; layer-wise probing was done on nine ImageNet-pretrained CNNs and on VGG16 and ResNet18 trained on CIFAR-10 and CIFAR-100. The authors suggest that because patch-level probing scales to deep CNNs, future work can use Shannon information estimators at the patch level to align the analysis more closely with the information bottleneck framework.
Recoverability is decoder-relative: in layers with a real bottleneck, wider decoders keep improving it, so the reported values are lower bounds and the authors do not claim they are saturated. Single-property random controls change one property at a time, so they show how recoverability is associated with a property rather than that the property causes it, and they do not cover interactions between properties or every property of pretrained weights. No single control explains learning-attributed recoverability across models, and eight candidate controls are scored per model, so a control that passes should be read as consistent with a mechanism rather than as proof of one.