Medical CLIP stops reading only high-level semantics: splitting texture and attribute branches lifts multi-abnormality accuracy by 10.2%
Synopsis
Medical CLIP has mainly aligned high-level semantic features, leaving texture, entity order and attributes poorly captured and limiting fine-grained tasks. Deco-CLIP splits both image and text into two branches: one ViT pretrained by CT-CLIP for semantics, one MAE-pretrained ViT for texture, with a text encoder that forks in its last three layers, shares keys and values, and uses distinct queries, plus threshold-adaptive sample mining and a regularized attention mechanism. On CT-RATE zero-shot tasks, multi-abnormality detection accuracy improves by 10.2% and report-to-volume retrieval recall by 16.2%.
Interpretation
Cross-modal alignment between medical images and radiology reports no longer rests on high-level semantics alone; texture, entity order and attributes are modeled separately. Earlier medical CLIP aligned image and text features with a single image encoder and a single text encoder, mainly separating high-level semantics, and performed poorly on texture, spatial relations and attribute recognition. Evaluations against CT-CLIP show the model fails to identify lesional liver images when non-lesional liver images are mixed with lesional image-text pairs, and fails to match correctly when the order of liver and lesion is reversed or the attribute terms for lobe and lung are swapped.
On the image side, two ViTs with different pretraining strategies handle semantics and texture; on the text side, the last three layers fork into two branches that share key and value vectors. The two text branches use distinct query vectors, producing attention maps with different distributions, and a regularized attention mechanism constrains the two maps to be orthogonal so each branch attends to different tokens. Attention map visualizations show the high-level branch assigns higher weights to sentence cores such as vascular structures, non-specific hypodense nodule and a 2 mm diameter nodule, while the other branch emphasizes attribute and spatial words such as natural, superior right lobe and of the left lung.
During training, semantically similar samples are adaptively clustered into mini-batches by intra-modal similarity, then secondary contrastive learning aligns fine-grained features. Conventional contrastive learning computes similarity over the whole batch, where fine-grained differences are easily swamped by many unrelated samples. With the similarity threshold set to 0.75, binary mask matrices are derived from the image-image and text-text similarity matrices, and the secondary contrastive loss is computed only over the selected samples.
Perspective
Evaluation uses the CT-RATE dataset, containing 3D chest CT volumes and corresponding radiology reports from 21,304 patients, with 20,000 patients for training and 1,304 for testing, and 18 abnormality types annotated per volume. Fine-tuning randomly selects 30% of the training set, 6,000 patients, with batch size 10 and learning rate 1.25e-6 in two stages. The method targets zero-shot multi-abnormality detection and report-to-volume retrieval, and the authors state the code will be publicly released.
The reported results are zero-shot outcomes on CT-RATE, where multi-abnormality detection accuracy improves by 10.2% and report-to-volume retrieval recall by 16.2%; the ablation shows that using the MAE image encoder with its text branch alone performs notably worse on both tasks, and only the combined branches reach the best performance.