CLIMB improves multimodal RAG with a confidence-guided complementary evidence pool, consistently beating retrieval-augmented baselines on Encyclopedic-VQA and InfoSeek
Related research and updates
Synopsis
The authors propose CLIMB, a training-free inference-time multimodal RAG framework that first builds a compact complementary evidence pool with an MMR-style objective balancing query relevance and passage-level redundancy, then performs confidence-controlled refinement within this fixed pool: an R/E/C critic scores passages by relevance, evidence specificity, and cross-modal alignment, while an evidence-grounded confidence estimator accepts an updated answer only when estimated confidence increases, yielding a simple stopping criterion; on Encyclopedic-VQA and InfoSeek it consistently improves over retrieval-augmented multimodal baselines, and ablations indicate complementary pooling, critic-based scoring, and iterative confidence-controlled refinement each contribute.
Figure 1: Comparison of multimodal RAG approaches. (a) Standard methods Cocchi et al. (2025) ; Caffagni et al. (2024b) ; Yan and Xie (2024) rely on Top-K similarity selection, which often produces redundant evidence with limited coverage and lacks explicit quality control for answer generation. (b) Our CLIMB framework addresses these limitations through two key innovations: diversity-aware evidence construction that balances relevance and complementarity to build a non-redundant passage pool, and confidence-controlled decoding that uses multi-faceted scoring and monotonic confidence constraints to ensure reliable answer updates.
arXivInterpretation
CLIMB is a training-free, inference-time multimodal RAG framework that brings external textual evidence into knowledge-intensive visual question answering without modifying the underlying retriever or MLLM. Compared with existing systems that rely on Top-K retrieval or reranking, CLIMB places the improvement in the inference procedure rather than in model or retriever training, reducing the deployment change surface. The abstract states the framework is training-free and inference-time and does not modify the underlying retriever or MLLM; implementation details are not expanded at the abstract level.
CLIMB builds a compact complementary evidence pool using an MMR-style objective that balances query relevance and passage-level redundancy. Addressing the redundant passages that Top-K retrieval may return, the method organizes the evidence set around complementarity rather than relevance alone. The abstract gives the design intent of this objective; ablations indicate complementary pooling contributes to final performance.
CLIMB performs confidence-controlled refinement within a fixed pool: an R/E/C critic scores passages by relevance, evidence specificity, and cross-modal alignment, and an evidence-grounded confidence estimator accepts an updated answer only when estimated confidence increases. Compared with limited control over whether an answer update is sufficiently supported by retrieved evidence, this design provides a simple stopping criterion and reduces unnecessary refinement. The abstract describes the mechanism and states it provides a stopping criterion; ablations indicate critic-based scoring and iterative confidence-controlled refinement each contribute.
On Encyclopedic-VQA and InfoSeek, CLIMB consistently improves over retrieval-augmented multimodal baselines. It reports consistent gains over existing retrieval-augmented baselines on two knowledge-intensive visual question answering benchmarks. The abstract reports consistent improvement; specific numbers, sample sizes, and statistical tests are not given in the abstract.
Perspective
The work targets knowledge-intensive visual question answering and suits teams that already have a retriever and a multimodal large language model and want to improve evidence use and answer updates without retraining; its design intent is confidence-controlled refinement within a fixed evidence pool, with rising confidence as the stopping condition. At the abstract level, evidence comes from the Encyclopedic-VQA and InfoSeek benchmarks plus component ablations, so its scope should be read as this class of visual question answering settings that require external textual evidence.
The abstract does not report specific performance numbers, sample sizes, or the concrete configurations of the retriever and MLLM, nor does it describe how the confidence estimator is implemented or how thresholds are set; the weighting of the R/E/C critic dimensions and the complementary pool size are not visible in the abstract. Readers who need to judge the size of the gains and statistical robustness still need the tables and experimental setup in the full text.