No labels on the target dataset: SAM, CLIP and a vision-language model team up for 60.8 AP zero-shot on COCO

A detector can be trained with no human annotation on the target dataset and still recognize both known classes and novel classes absent from training. Prior open-vocabulary detectors still depend on closed-set detectors pretrained on human-annotated data, while unsupervised discovery methods cannot assign open-ended semantic labels. Under zero-shot settings on COCO val2017 and LVIS minival, the YOLO-backbone variant reaches 60.8 AP and 47.4 AP, and the Swin-T variant reaches 57.2 AP and 46.2 AP.

The pipeline splits labeling into three steps: a segmentation model yields class-agnostic boxes, a vision-language model names the scene and objects, and image-text matching attaches names to boxes. Querying a large vision-language model per proposal is expensive, so the smaller CLIP model pre-filters and matches labels, and only cluster medoid images go to the large model. Landmark-based clustering cuts per-image labeling time from 15–20 minutes to 45 seconds, roughly a 20–27 times speedup.

The trained detector then supplies box prompts back to the segmentation model, refining masks and localization, and the detector is retrained on the improved pseudo-labels. Without prompts the initial segmentation model emits background, redundant parts and noisy masks that must be filtered before serving as training targets. On ODinW-35 and Roboflow-100 validation sets, the YOLO-backbone variant reaches 32.5 AP and 23.7 AP, and the Swin-T variant reaches 30.1 AP and 20.8 AP.

The demonstrated scope is zero-shot evaluation on COCO val2017, LVIS minival, ODinW-35, Roboflow-100 and a COCO open-vocabulary split, plus one domain-shift dataset built with the Applied Intuition simulator; foundation models are used only in offline labeling, and the deployed detector is a YOLO-type model.

Mislabeling of ambiguous or occluded objects by the large vision-language model propagates into the pseudo-labels; localization quality is bounded by the segmentation model and degrades under low contrast, clutter or fine-grained overlapping parts; labeling is sensitive to prompt phrasing, only partly mitigated by the label dictionary; and the label-quality trade-off of clustering while lowering query cost remains uncharacterized.