MEOW turns frozen 3D segmentation models into open-world learners: no parameter updates, yet it remembers structures never seen in training
Synopsis
MEOW converts non-interactive 3D segmentation backbones into open-world interactive segmentation systems without updating any learnable parameters. Interactive segmentation had been confined to the open-set setting, able to predict or reject unknown structures but not to accumulate those predictions. MEOW pairs a frozen feature extractor with parameter-free transductive reasoning and memory consolidation, storing user-confirmed pseudo-labels in memory for later queries to retrieve; on classes never seen in training it keeps improving its own predictions as memory grows, and it transfers zero-shot to micro-CT and other imaging modalities.
Interpretation
MEOW gives interactive segmentation open-world capability for the first time: during deployment the system writes user-confirmed predictions of unknown structures into memory, so later queries of the same kind benefit directly while the backbone's learnable parameters stay frozen. Interactive segmentation had remained in the open-set setting, predicting or rejecting unknown structures without turning that prediction into knowledge for the next query; open-world learning had appeared in classification, object detection and semantic segmentation, but not in interactive segmentation. On classes never seen in training, MEOW consistently improves its own predictions as memory samples grow; it transfers zero-shot to unfamiliar domains on the micro-CT Mouse CT and InsectAnatomy data and the MRI brain tumor MSD BRATS data, and it outperforms VISTA3D by a large margin in the open-world setting.
The system splits segmentation into three parts: a frozen multi-scale feature extractor for perception, a memory module holding confirmed samples, and a parameter-free transduction module that re-retrieves and predicts at every interaction. The transduction module replaces the usually trained decoder with approximate nearest-neighbor retrieval, searched level by level from global to local over the feature pyramid, followed by hierarchical message passing to keep spatial consistency. Experiments cover organ CT, Mouse CT, InsectAnatomy and MSD BRATS, with feature extractors taken from U-Net SuPreM, TotalSegmentator and VISTA3D's automatic branch; writing or deleting a memory sample takes milliseconds, so learning and unlearning are equally fast.
Perspective
The demonstrated scope is interactive segmentation on 3D volumetric data: known organ classes in CT, unknown classes in micro-CT and insect anatomy, and cross-modality transfer to MRI brain tumors; experiments ran on individual RTX 3090, A40 or A100 GPUs, with timing measured on a single workstation with an RTX 3090.
Storing feature pyramids and running nearest-neighbor search over memory contents requires more GPU memory and slightly higher inference time than fully end-to-end neural approaches; the system requires hourglass-shaped feature extractors and currently cannot use transformer architectures for open-world learning.