Remote-sensing fusion stops optimizing pixels alone: Lance closes a text loop and lifts detection mAP by 6.4 on a held-out detector
Synopsis
Remote-sensing image fusion and semantic interpretation have usually been optimized separately, so good fusion metrics need not preserve downstream retrieval or detection utility. Lance maps each input to a closed-vocabulary text record of scene, objects and layout, conditions fusion on that language, re-describes the fused image as text, and back-propagates through text consistency and an anti-collapse text–mask contrastive loss with negative queries. Across seven test sets on GF2, QuickBird, WorldView-III and WorldView-II, reduced-scale reference metrics improve by up to 11.7% (median 2.1%) over the strongest learning baseline, and a held-out RemoteCLIP verifier plus a supervised DOTA detector give +6.4 mAP@0.5 over the closest competitor.
Interpretation
A fused remote-sensing image now has to satisfy pixel fidelity and a language-queryable semantic consistency at the same time, not just low spectral and spatial distortion. Fusion and semantic interpretation were previously trained separately, and fusion objectives neither rewarded downstream describability nor checked whether the output kept the input's semantic content. On WorldView-III fused outputs, methods whose spectral fidelity is statistically indistinguishable differ by more than 6 mAP points on an independent aerial-object detector, a gap fidelity metrics do not capture.
The loop has three stages: the input is projected to pseudo-RGB and matched to a structured text record, the fusion network injects that language at decoder levels 2 to 4, and the fused image is re-described as text so the two descriptions can be compared. The re-description step replaces hard thresholds and discrete connected-component counting with a differentiable continuous summarizer whose soft area, soft count and centroid–dispersion descriptors let gradients reach the fusion generator. On WorldView-III the full loop reaches Q2n 0.9166 and HQNR 0.9439, and both metrics rise monotonically as text conditioning, the dense interpreter, the continuous summarizer and the two semantic losses are added in turn.
Collapse of the semantic signal is prevented by a negative-query term: besides positive queries, 8 hard negatives are sampled per image so a constant all-present mask becomes costly. With a positive-only text–mask loss the average soft area inflates from 0.18 at epoch 50 to 0.71 at epoch 1000, the classic collapse signature. Adding the negative term keeps the soft area within 0.14 to 0.24 throughout training, consistent with the gradient-sign analysis for a constant mask.
Perspective
The demonstrated scope is seven test sets on four sensors, GF2, QuickBird, WorldView-III and WorldView-II, with WorldView-II held out for cross-sensor generalization; semantic evaluation uses RemoteCLIP and a DOTA supervised detector unseen during training, the fusion backbone costs 22.9 GFLOPs and 0.42M parameters, and the full pipeline adds 14 ms of language processing at 256×256.
The system is closed-vocabulary, with its state restricted to 12 scenes, 32 objects and 8 layouts, and the open-vocabulary F1 measures transfer to a held-out evaluator rather than open-set recognition; missing categories map to their nearest vocabulary entry, and frozen vision–language model coverage remains a limit. In the per-class breakdown vegetation shows a small regression of −0.009 because the closed-vocabulary text cannot separate forest, farmland and grassland.