Two encoders no longer share one space: DuoJEPA adds a single translator layer and lifts chest-radiograph zero-shot AUC from 60.0 to 86.6
Synopsis
Paired modalities are usually used to pull two encoders into one shared embedding space. DuoJEPA instead keeps a separate LeJEPA self-supervised space per modality and appends to each encoder one residual-free transformer layer that translates its embedding into the other encoder's space, trained by a contrastive loss computed entirely within that target space. On CheXpert Plus chest radiographs and reports it reaches a zero-shot AUC of 86.6 and a cross-modal retrieval score of 78.5, against 60.0 and 64.2 for a reimplementation of LeVLJEPA with the same encoders and data, while a paired-cosine test detects no direct alignment between the two own spaces.
Interpretation
Paired-modality pre-training no longer requires the two encoders to map a sample to the same coordinates; cross-modal retrieval and zero-shot classification come from a learned translation layer instead. Contrastive alignment previously trained two encoders until the embeddings of a pair coincided, and non-contrastive methods such as LeVLJEPA still kept a shared prediction target. On CheXpert Plus chest radiographs and reports, DuoJEPA reaches a zero-shot AUC of 86.6 and a cross-modal retrieval score of 78.5, against 60.0 and 64.2 for a reimplementation of LeVLJEPA with the same encoders and data, with paired bootstrap intervals excluding zero.
Each modality gets its own LeJEPA-trained self-supervised encoder, followed by one transformer translator layer that maps that modality's embedding into the other encoder's space. The translation loss is an InfoNCE contrastive objective computed only inside the target space, with no term comparing the two own spaces directly, yet its gradient also reaches the target encoder, so the encoders are trained jointly. On Sentinel-2 optical and Sentinel-1 radar imagery, the translator retrieves the paired image among 4,000 candidates in 77.3% of cases from optical to radar and 79.2% from radar to optical, whereas the two own spaces retrieve each other's images in only 0.45%.
Whether the translator keeps its residual connections decides whether the cross-modal pressure is absorbed by the translator or by the two encoders. A standard transformer layer adds its output to its input, so a translator built from it starts close to the identity map and the loss can be reduced by moving the two encoders together. With a residual translator, the paired-cosine alignment AUC for radiographs and reports rises to 97.5 and the two satellite own spaces retrieve each other in 64.7% of cases, against at most 0.53% for every skipless configuration, at comparable cross-modal performance.
Perspective
The results cover two modality pairs: chest radiographs with reports from CheXpert Plus, and Sentinel-2 optical with Sentinel-1 radar imagery pre-trained on SSL4EO-S12 and evaluated on BigEarthNet; the radiograph model is trained for 20k steps and the satellite models for 100k steps, each as a single run. The intended readers are researchers and engineers working on multimodal representation learning and on medical imaging or remote-sensing applications.
The alignment verdict rests on a paired-cosine test, and an AUC near 50 shows only that this test detects no direct alignment, not that the two spaces are unrelated; on satellite imagery the trained translator retrieves 4 to 6 points below a rotation fitted on 8,000 pairs, a gap outside the bootstrap intervals whose cause was not tested.