CroCoMAE moves SAR-optical alignment from whole images to patches, with the largest gains on semantic segmentation
Synopsis
SAR and optical remote sensing images are often treated as largely redundant views, and earlier cross-modal self-supervised methods aligned them only at the whole-image level, leaving fine-grained complementary structure unused. CroCoMAE combines image-level contrastive learning with patch-level masked reconstruction inside a ViT, splitting each patch embedding into a shared subspace aligned across sensors and an unconstrained unshared subspace. On the EuroSAT-SAR scene-classification and DFC2020 semantic-segmentation benchmarks it beats uni-modal self-supervised baselines and retrained cross-modal frameworks in most configurations, with the largest gains on segmentation.
Interpretation
Cross-modal self-supervised pretraining for SAR and optical imagery moves from whole-image alignment to alignment between patches at the same spatial position, so fine-grained complementary information within one scene can be represented separately for the two sensors. Earlier cross-modal frameworks aligned the two modalities mainly through global image-level representations, or treated the two views as largely redundant, even when cross-modal contrast was combined with masked reconstruction; patch-level cross-modal alignment was underexplored. On the EuroSAT-SAR scene-classification and DFC2020 semantic-segmentation benchmarks, compared with uni-modal self-supervised methods and cross-modal frameworks including DeCUR and CROMA retrained under the same data, ViT-Small backbone and fine-tuning protocol, CroCoMAE gives the best result in most configurations, and its gains exceed one standard deviation over five runs in most configurations.
Each modality's encoder splits its patch embedding in half, using one half for cross-sensor alignment and leaving the other unconstrained to retain sensor-specific information. The shared/unshared split follows the subspace-decomposition idea, but applies it to spatially localised ViT patch tokens instead of global image embeddings, while reconstruction stays independent per modality. Direct measurement on frozen encoders shows co-registered SAR and optical patches have mean cosine similarity of 0.96 (DFC2020) and 0.99 (EuroSAT-SAR) in the shared subspace, versus 0.00 for random pairs; in the unshared subspace they reach only 0.01, comparable to random pairs.
An image-level contrastive loss on the class token keeps global representations invariant to radiometric and geometric augmentation, while patch level is driven jointly by masked reconstruction and cross-modal contrast. The three losses are summed without weights, so no loss-balancing hyperparameters or per-task tuning are needed. On EuroSAT-SAR the full model gives the highest average performance in the optical and multi-modal settings, for example 96.5/96.0 and 96.4/95.3 at 1% labels; removing the image-level contrast term gives the strongest SAR scores but degrades optical and multi-modal results.
Perspective
Evaluation covers two SAR-optical benchmarks, EuroSAT-SAR scene classification and DFC2020 semantic segmentation, using Sentinel-1 dual-polarisation SAR and Sentinel-2 thirteen-band optical imagery with a shared ViT-Small backbone; EuroSAT-SAR uses a transductive setting where pretraining and fine-tuning share images, while DFC2020 pretrains on the training split and fine-tunes and evaluates on the disjoint validation split. The authors list extension to non-remote-sensing modality pairs such as RGB-Depth or thermal, and more effective combination of image-level and patch-level representations, as future work.
Performance is largely insensitive to the shared-subspace ratio between 10% and 70%, staying within about one standard deviation, while above 70% SAR performance drops noticeably, indicating alignment needs little shared capacity but too little left for the unshared subspace loses modality-specific information. The default 50/50 split lies in the stable range and is at most 0.7 points from the best configuration, which the authors position as a reasonable dataset-agnostic choice rather than a per-benchmark optimum. Pretraining costs more than CROMA, 34 versus 10 GPU-hours on DFC2020, with the extra cost coming from full-image forward passes and dense patch contrast.