Alibaba DAMO's HoloCT fuses CT reading and 3D segmentation in one model, surpassing all four radiologists in sensitivity and overall accuracy in a multicentre reader study

A CT reading and its 3D localization now come from one model: HoloCT can attach a 3D mask of the corresponding anatomical region or finding to its text, so a reader can inspect the evidence slice by slice. Segmentation networks delineated organs and abnormalities at the voxel level without interpreting their clinical significance, while CT vision–language models generated diagnostic text without voxel-level localization, leaving the anatomical basis of predictions hard to inspect. Across 11,455 clinical CT scans in six cohorts at five hospitals, zero-shot HoloCT-8B reached a mean Dice of 0.488 (95% CI 0.441–0.531) on laryngeal lesions versus 0.107 (0.066–0.150) for VoxTell, and 0.715 (0.687–0.741) versus 0.543 (0.506–0.579) on lung cancer.

The model feeds whole-volume context and fine-grained anatomical detail into the language model together, so clinical reasoning can draw directly on spatial features shared with segmentation. Existing grounded medical models connected language and segmentation only at the output level, routing high-resolution spatial features to the mask decoder rather than the language model, and some still relied on 2D slice encoders or coarsely downsampled 3D inputs. HoloCT uses a global encoder producing 256 whole-volume tokens and a detailed encoder producing one fine-grained token per voxel at 1.5 mm isotropic spacing, both supplied to a Qwen3-VL backbone; segmentation tokens from the language model then drive a mask decoder. Across 346 tasks on 61 internal and 12 external datasets, a single HoloCT-8B reached a mean Dice of 0.816 over 272 internal categories, above 0.799 for a collection of 82 dataset-specific nnU-Net models.

Text and spatial abilities reinforce each other during training and inference rather than sitting side by side as separate outputs. It remained an open question whether clinical understanding and spatial perception could be learned jointly within a single foundation model and mutually reinforce one another. With only fine-grained tokens supplied to the language model, adding language supervision to segmentation-only training raised Dice from 0.601 to 0.613, and adding segmentation supervision to language-only training raised screening macro F1 from 0.256 to 0.351; at inference, masking language-model attention to fine-grained tokens dropped segmentation Dice from 0.840 to 0.811 and screening macro F1 from 0.678 to 0.061.

The demonstrated scope is retrospective: 73 public datasets plus 11,455 clinical CT scans in six cohorts at five hospitals, covering referring segmentation, lesion screening, visual question answering and report generation; finding-level 3D masks are currently limited to chest CT, and hospital evaluation was conducted within a single country.

The authors note that segmentation accuracy on musculoskeletal bone lesions and diffuse, non-focal chest findings remains limited; the model interprets single CT volumes independently, without longitudinal comparisons, multi-phase contrast studies or electronic health records; and clinical translation still requires prospective validation across international cohorts, reader-in-the-loop trials of interactive 3D mask verification, and calibrated uncertainty estimation with automated quality control.