Composed image retrieval stops squeezing queries into one vector: FactEmbed decomposes the edit first, then reranks candidates
Synopsis
Composed image retrieval takes a reference image plus a modification text and must return the target image, and the common recipe compresses that multimodal query straight into one retrieval vector, leaving the model no explicit chance to work out what the text changes. FactEmbed inserts a set of learnable factorized latents into Qwen2.5-VL-7B's input and trains them with a target-imagination objective that reconstructs target details, so each latent grounds a semantic region, then carries that decomposition into retrieval through a factor-level contrastive loss and attention distillation. On the refined CIRR and FashionIQ benchmarks it scores above both CSMCIR and ReCALL on Recall@1 and average Recall@10, though the improvement margins stated in the text do not match the table values.
Interpretation
Composed image retrieval now separates query comprehension from discriminative embedding: the model first simulates the visual edit the text requests, then builds the retrieval representation on top of that decomposition. Earlier MLLM methods mapped the multimodal query directly onto a single embedding under contrastive supervision alone, with no explicit step resolving which visual attributes the text modifies. On refined CIRR, FactEmbed reaches 68.99 Recall@1, above CSMCIR at 67.16, which additionally uses synthetic captions for gallery images, and above ReCALL at 67.41; the text separately claims gains of 2.7% and 2.3% over the two, which does not match the table values.
The decomposition is carried by a set of learnable latents placed before the [EOS] token, acting as an information bottleneck that splits the fused multimodal input into factors. These latents gain structure without explicit reasoning traces or dense annotations, learning their division of labor by reconstructing target patch features. A lightweight decoder attaches at an intermediate layer of the 28-layer MLLM and is conditioned on a coarse layout prior average-pooled to low resolution, so the latents only supply detail the prior cannot explain; without that prior they collapse into a few coarse regions.
Retrieval training uses two contrastive levels and distills the decoder's spatial decomposition into deeper attention layers of the MLLM. Reconstruction alone only makes the latents useful for generation and does not guarantee that query and target land in comparable coordinate spaces. A factor-level InfoNCE matches query and target factor sets through optimal transport while an instance-level loss optimizes the [EOS] embedding; at inference [EOS] retrieves first and mean-pooled latents rerank only the top 100 candidates, adding 0.17 ms per query at batch size 1 on the CIRR validation set.
Perspective
Evaluation covers the refined CIRR and FashionIQ composed image retrieval benchmarks, with original CIRR and FashionIQ results also reported; the default backbone is Qwen2.5-VL-7B-Instruct, with additional runs on Qwen3-VL-8B and Qwen3-VL-Embedding-8B. The intended readers are researchers and engineers working on multimodal retrieval and embedding learning.
The improvement margins stated in the text over CSMCIR and ReCALL do not match the table values, so specific gains should be taken from the table or the table values reported directly. The number of latents is fixed, and no explicit supervision specifies how many semantic factors there should be; full-factor matching is more accurate, but inference reranks with mean-pooled latents and gives up part of the fine-grained information; evaluation is limited to CIRR and FashionIQ, so generalization as a unified multimodal retriever still needs verification. When contrastive captioning scales up hard negatives, sibling hard negatives mined for the same query can still overlap in semantics.