Seafloor classifiers stop leaning on color and texture alone: GHiML adds VLM-derived semantics, lifting macro F1 by up to 21.1 points on in-domain benthic data
Synopsis
Hierarchical multi-label classification of seafloor imagery has relied on low-level visual features such as color, texture and shape, leaving rare classes and shifting illumination hard to resolve. GHiML extracts concept variables tied to the training data from a vision language model, generates per-sample captions grounded in them, and fuses the resulting text embeddings with image embeddings through cross-attention and a per-node gated router. Across BenthicNet, FathomNet and iNaturalist, the router improves macro F1 over the vision-only baseline by up to 6.7 percentage points, and by up to 21.1 points on in-domain benthic data, with gains persisting under distribution shift.
Interpretation
Hierarchical multi-label classification of seafloor imagery gains a semantic channel drawn from a vision language model that runs alongside image features, rather than having the VLM emit species labels directly. Vision transformers trained on these tasks relied mainly on low-level visual latents such as color, texture and shape, lacking the higher-level reasoning humans use for identification, while zero-shot VLM calls could not produce scheme-conformant predictions. On BenthicNet, FathomNet and iNaturalist, the router models are strongest in nearly all settings, improving macro F1 over the vision-only baseline by up to 6.7 percentage points and by up to 21.1 points on in-domain benthic data, with the method ranking consistent across random seeds.
The pipeline first treats the vision language model as a source of concepts, asking it to propose semantic variables relevant to the label hierarchy and the training data, then captions each image with those variables. The variables are required to differ from low-level visual latents, and are refined by expectation-maximization disambiguation, smoothing toward the prior and KL-divergence scoring, typically leaving 15 to 20 variables per domain. Images are embedded with a DINOv3 ViT-H backbone and captions with Qwen3-Embedding-8B; cross-attention uses images as queries and text as keys and values, and a gated router outputs per-node, per-sample softmax weights over the three modalities, all without fine-tuning the VLM or synthesizing training images.
The text contribution acts mainly as a coarse prior over regions of the hierarchy rather than as finer discrimination within a subtree. Among 171 BN-Bio nodes with sufficient support, the text branch scores higher on the inverted HML score while failing to identify the node on macro F1 at 75 nodes, against only 6 nodes where the reverse holds. Replacing the per-sample text signal with a mean vector lowers the inverted HML score on BN-Bio while leaving macro F1 unchanged, consistent with a change in prediction cardinality, whereas single-label iNaturalist loses 18.7% of macro F1.
Perspective
The demonstrated scope is hierarchical multi-label classification on BenthicNet's substrate and biota tasks, FathomNet and iNaturalist, covering in-domain splits and out-of-domain splits formed by geospatial partitioning or hand selection; the method targets domain-specialized labs running on consumer-grade GPUs.
Variable and caption quality is bounded by the VLM's pretraining corpus, so understudied groups such as soft corals may yield shallower or inaccurate variables; the correlations between variables and text advantage are observational and do not establish a causal mechanism for any single variable; the inverted HML score treats all hierarchy edges as unit weight and its cardinality penalty is governed by a user-set parameter.