Multi-modal fusion stops trading away semantics: an attack-defense game overlays a semantic map, lifting segmentation mIoU by 1.5
Synopsis
Multi-modal image fusion has long traded visual quality against semantic attributes because the two optimization goals conflict. AD-SRF separates them: a transformer-based restorative visual fusion network builds a clear baseline, then an attack-defense game has an attacker mine semantic weaknesses while a defender generates an imperceptible semantic map overlaid back onto the fused image. On MFNet and FMB, it ranks first in four and five of six visual metrics, with segmentation mIoU 1.5 and 1.73 above the runner-up.
Interpretation
Multi-modal image fusion can now optimize visual quality and semantic attributes separately, then merge them into one image instead of trading one for the other. Earlier fusion methods either optimized visual appearance alone and ignored the semantics downstream segmentation needs, or forcibly coupled segmentation and fusion models in joint optimization, compromising both. On MFNet and FMB, the method ranks first in four and five of six visual metrics, and its segmentation mIoU exceeds the runner-up by 1.5 and 1.73.
The procedure has two steps: freeze a fusion network that handles vision only, then let an attacker and a defender take turns modifying that image. The attacker learns a perturbation pattern that specifically breaks downstream segmentation, while the defender restores lost semantics from the original inputs and overlays them back through an imperceptible semantic map. Training uses 784 training and 393 testing pairs from MFNet and 1,220 training and 280 testing pairs from FMB, with OverLock-B as the segmentation backbone and only its auxiliary head and decoder updated.
The attack-defense mechanism can be attached to other fusion methods as a semantic enhancement module. ADG was deployed on five existing fusion methods, taking the fused images they originally produced as input and refining them through attack-defense training. Across the five fusion methods, adding ADG produced consistent segmentation improvements.
Perspective
The method was validated on MFNet and FMB for simultaneous gains in visual quality and segmentation semantics, and extended zero-shot to polarization image fusion and NIR-VIS fusion. Segmentation evaluation uses the OverLock-B backbone, and the attack-defense training stage updates only its auxiliary head and decoder. Reinforcement currently targets global semantics, and the authors propose using vision-language models with language instructions for finer, controllable semantic reinforcement.
The authors note that the method focuses on global semantic reinforcement and may lack the flexibility to handle specific objects that need extra attention, which they state as an open problem. Attack-defense training uses alternating optimization, cycling between an attack phase and a joint defense phase, with the segmentation network's learning rate scaled by a multiplier to keep updates stable.