Cross-modal remote sensing retrieval stops settling for one point estimate: CAPE-JEPA lifts F1@5 by 13.74 points on BEN-14K cross-modal directions
Synopsis
Cross-modal remote sensing retrieval must match semantically identical scenes between synthetic aperture radar and multispectral imagery, yet earlier JEPA-style methods regress a single deterministic latent for each masked patch and assume every region is equally predictable. CAPE-JEPA instead outputs a per-patch Gaussian mean and variance, trains it by negative log-likelihood, and adds adversarial cross-modal distributional regularization plus in-training contrastive learning. On BEN-14K and fMoW it beats baselines including X-JEPA, SkySense and CROMA across all eight retrieval directions, gaining up to 13.74 points over X-JEPA on BEN-14K cross-modal directions and up to 15.59 on fMoW, with 407.30M parameters versus 432.72M.
Interpretation
Cross-modal remote sensing representations can now carry per-patch predictive uncertainty instead of compressing each masked patch into one deterministic vector. The earlier X-JEPA regressed a single latent point estimate per masked patch with a cross-attentional predictor, implicitly treating all regions as equally predictable. On BEN-14K, CAPE-JEPA exceeds X-JEPA by 13.74 and 13.52 F1@5 points on the S1S2 and S2S1 cross-modal directions, and on fMoW by 8.16 and 15.59 points on RGBSEN and SENRGB.
The mechanism writes latent prediction as a conditional Gaussian and pulls the two modality embedding distributions toward each other. The predictor emits a mean and variance for each masked patch and is trained by negative log-likelihood, so the variance term down-weights ambiguous regions such as clouds, radar speckle and mixed land-cover boundaries. The cross-modal regularizer XSIGReg uses Epps-Pulley characteristic-function tests to find and penalize the projection directions of largest distributional divergence without explicit sample pairing; the full objective gives the highest F1@5 on all eight directions of BEN-14K and fMoW.
Distributional alignment alone erases class separation, so scene-level contrastive supervision runs during training. The ablation shows S2S1 dropping 2.03 points with XSIGReg but no InfoNCE, and dropping 8.89 points with InfoNCE but no XSIGReg. With all three losses enabled, BEN-14K reaches 79.22, 80.64, 76.48 and 76.50 on S1S1, S2S2, S1S2 and S2S1, above any configuration missing one term.
Perspective
The results cover SAR, multispectral and RGB modalities on the BigEarthNet-14K and fMoW-Sentinel datasets, report F1@5, and are evaluated on the validation sets under the established protocol. The method targets cross-modal remote sensing image retrieval, and its readers are researchers and engineers working on multi-sensor representation learning and retrieval system design.
Complementary masking underperforms independent masking on the S2S1 direction alone, ViT-L underperforms ViT-B on three of four directions, and extending alignment beyond 25 epochs drops S2S1 from 76.18 to 66.92. Failure cases concentrate on spectrally ambiguous land-cover boundaries where several semantic classes coexist at patch scale.