Xiaomi's 4B multimodal model sends mask supervision into the shared backbone, gaining 5.6 points on referring comprehension
Synopsis
Xiaomi released Xiaomi-Lens-0, a 4B multimodal model that decodes region masks from its own visual representations instead of relying on an extra segmentation encoder. Earlier multimodal models either took spatial detail from a separate segmentation encoder or kept the vision encoder frozen, so mask loss never reached the shared features. Xiaomi-Lens-0 pairs vision-encoder patch features with visual latents, the language model's states at generated mask tokens, in a lightweight decoder, so pixel-level supervision reaches the backbone shared with language generation; it gains 5.6 points on average over Qwen3.5 across all eight RefCOCO, RefCOCO+, and RefCOCOg splits and reaches the highest segmentation cIoU among compared models.
Interpretation
A multimodal model can now decode region masks from the same visual features it uses for language generation, so mask loss updates the shared backbone without a separate segmentation encoder. Earlier designs split two ways: LISA and Sa2VA rely on an additional hierarchical segmentation encoder, and Qwen3-VL-Seg fuses multi-scale features from a supplementary image encoder; PixelLM and MLLMSeg decode from the model's own visual features but keep the vision encoder frozen, so mask loss does not reach shared visual features. Across all eight RefCOCO, RefCOCO+, and RefCOCOg splits, referring comprehension gains 5.6 points on average over the Qwen3.5 backbone, with the largest mean gain on RefCOCO+ at 7.6 points; on all eight RES splits it reaches the highest cIoU among compared models, averaging 81.4 against 78.6 for the strongest baseline, Qwen3-VL-Seg.
During reasoning the model can write text and emit masks for the regions it mentions, so intermediate regions cited in a chain of thought also receive pixel-level supervision. Previously mask supervision mainly served final segmentation targets, and regions cited at intermediate reasoning steps carried no explicit extent annotation. The model takes the patch grid from before the vision encoder's spatial merger and feeds it, with the language model's contextual state at a special mask token (the visual latent), into a lightweight decoder; training uses grounded reasoning traces that pair intermediate region references with masks. Ablations show that adding these traces to a grounding-only setting raises REC by 0.1 to 0.6 points and RES by 0.5 to 0.8 points on the three validation splits.
Perspective
The evaluation covers six tasks: referring expression comprehension, referring expression segmentation, multi-instance grounding, reasoning segmentation, embodied referring, and fine-grained perception, with main-model results averaged equally over three Stage 2 checkpoints. The authors also build LensBench, spanning general, aerial, driving, and panoramic imagery with 300 complete evaluation units per domain, 1,200 paired units and 4,800 instruction-level samples, each unit human-reviewed for target completeness and instruction-mask agreement. Readers should treat the numbers as the behavior of a 4B model on the stated benchmarks and the new benchmark, not as a general conclusion across scales or backbones.
Implicit instructions remain clearly harder than explicit ones: on LensBench, implicit cIoU is 54.7, 16.7 points below explicit, ranking fifth; implicit selection of multiple targets in panoramic scenes is especially difficult, with Prec@0.5 falling from 81.3 for explicit to 48.7. On general imagery Qwen3.8-Max leads at 74.2 cIoU against 63.9 for Xiaomi-Lens-0. The multi-instance grounding lead comes from tighter boxes at strict thresholds rather than from recovering more instances.