CLIMB improves multimodal RAG with a confidence-guided complementary evidence pool, consistently beating retrieval-augmented baselines on Encyclopedic-VQA and InfoSeek
The authors propose CLIMB, a training-free inference-time multimodal RAG framework that first builds a compact complementary evidence pool with an MMR-style objective balancing query relevance and passage-level redundancy, then performs confidence-controlled refinement within this fixed pool: an R/E/C critic scores passages by relevance, evidence specificity, and cross-modal alignment, while an evidence-grounded confidence estimator accepts an updated answer only when estimated confidence increases, yielding a simple stopping criterion; on Encyclopedic-VQA and InfoSeek it consistently improves over retrieval-augmented multimodal baselines, and ablations indicate complementary pooling, critic-based scoring, and iterative confidence-controlled refinement each contribute.