Lesion segmentation drops the per-lesion box: one concept prompt lets SAM3 outline 11 lesion types across 13 datasets

Lesion segmentation moves from drawing a box around each instance to giving one concept, and the model outlines every lesion of that kind at once. SAM and SAM2 accepted only geometric prompts such as points or boxes, so multiple lesions of the same type in one image each needed their own prompt. Across 13 datasets and 11 lesion types, SAM3 exceeded SAM2 on Dice in 10 datasets, by 4.41 on ISLES22, 3.04 on MSD Pancreas and 5.39 on ISIC2018, while SAM2 held a sub-1% Dice edge only on BraTS2023-MET, MSD Lung and BUSI.

A concept prompt has two parts, text and an image exemplar, and using both gives the best localization and boundaries. With text alone, Dice was extremely low on several datasets: 0.69% on BraTS2023-MET, 3.96% on MSD Pancreas, 9.17% on MSD Colon and 8.71% on KiTS2019. Adding an image exemplar improved every dataset, and combining text with the exemplar added a further 2.90, 3.07, 2.72 and 11.54 Dice on BraTS2023-MET, MSD Lung, MSD Pancreas and KiTS2019, improving HD95 and ASD across all datasets, with only ISLES22-DWI losing a little Dice.

Concatenating prior information into the input channels raises segmentation performance without extra training. These priors had not been used as input channels for SAM3. On three in-house adaptive-radiotherapy datasets, adding the previous fraction's MR image and annotation raised Dice by 10.79, 7.87 and 4.47 on Mix-Seq-Brain, One-Seq-Liver and Mix-Seq-Abdomen, and on BraTS2023-MET combining T1-C, T2-F and T2-W gained 2.78 over T1-C alone.

When fine-tuning SAM3, training only lightweight adapter modules is steadier than updating all parameters or the detector alone. Full fine-tuning and detector-only fine-tuning gave limited or inconsistent gains over the un-fine-tuned baseline. On BraTS2023-MET, adapter-based fine-tuning reached the highest Dice of 82.96% and the lowest ASD of 0.9; applying that strategy uniformly to five public datasets gave LeSAM3 higher Dice on BraTS2023-MET, ISLES22, ISIC2018 and Kvasir-SEG but degraded performance on MSD Pancreas.

The report covers 13 datasets and 11 lesion types over MRI, CT, ultrasound, dermoscopy and endoscopy, with emphasis on volumetric data; the three prior-input enhancements were evaluated separately, not jointly. Image exemplars were drawn from the same test slice, so the pipeline assumes the location of at least one lesion is known in each test slice. Adapter-based fine-tuning performed best on BraTS2023-MET, and a unified adaptation strategy trained on five public datasets did not improve every task.

Text-prompt performance depends heavily on wording: on MSD Lung, 'lesion' scored 62.36% Dice while the semantically similar 'tumor' scored 0.00%, and 'lung lesion' scored 61.63% against 6.18% for 'lung tumor'. A preceding-slice prediction mask as prior did not help every dataset, degrading MSD Lung and improving Dice on Mix-Seq-Brain while worsening HD95. Multimodal concatenation also varies with dataset and modality quality, and on ISLES22 the combined input scored lower Dice than DWI alone.