Vision-language hallucinations get a fix at the encoder: three LoRA experts, no retraining per model
Synopsis
Hallucinations in large vision-language models partly trace to CLIP-style encoders that favor main objects and skip image detail. PLANE attaches three LoRA visual experts to CLIP, each trained on object, attribute or relation captions, and an instruction-aware gate fuses the four image representations at token level. Trained once on LLaVA-1.5 7B, it plugs into other models sharing the same encoder; on LLaVA-1.5 7B the MME hallucination subset total rises from 555.38 to 582.34 and the GIFT hallucination rate falls from 23.39% to 20.30%.
Interpretation
One route to fewer vision-language hallucinations moves the change to the vision encoder rather than adding a decoding strategy or a second model that rewrites the output. Earlier work falls into three groups: VCD and OPERA intervene at inference and raise inference time; Volcano and LURE use a revisor model to correct generated text, which slows the final output and leaves the internal cause untouched; LRV fine-tunes on more robust datasets and usually needs retraining for each model. On the MME hallucination subset, adding PLANE raises the total from 555.38 to 582.34 for LLaVA-1.5 7B, from 560.54 to 586.11 for Mini-Gemini 7B, from 626.12 to 647.78 for Parrot 7B, from 445.67 to 464.67 for Otter 7B, from 433.66 to 471.34 for Vary Toy 1.8B, and from 300.01 to 319.67 for LLaVA 13B.
The three experts split the image into objects, attributes and spatial relations, and the fusion weights shift per token with the user's instruction, so one module can draw on different visual aspects for different questions. The three LoRA experts are lightweight adapters trained on CLIP, each contrastively aligned on captions from one aspect and additionally trained against deliberately constructed hallucinated captions as negatives; the Ins-Gate gate network combines a cross-attention layer and a feed-forward network and outputs a weight per image token over the four representations. The training set holds 38,375 samples: 7,800 image-object caption pairs, 18,121 image-attribute pairs and 12,454 image-relation pairs; all original vision-language model parameters stay frozen, with experts trained in stage one under the CLIP paradigm and only the Ins-Gate trained in stage two.
The accompanying GIFT benchmark takes hallucination judging away from GPT-4 and hands it to Llama-Judge, trained on Llama 3. Among earlier generative hallucination benchmarks, CHAIR and AMBER cover only the object level, while Faithscore spans multiple levels but depends on ChatGPT as judge, which raises cost and limits scaling. GIFT selects 206 images from the DCI validation and test sets and manually annotates 2,120 scene-graph triplets; on Vary Toy, Llama-Judge agrees with human judgments on 83.70% of triplets against 71.81% for GPT-4.
Perspective
The demonstrated setting is vision-language models that share the same CLIP vision encoder: experts are trained on CLIP-ViT-L-224 and CLIP-ViT-L-336, and PLANE is trained only on LLaVA-1.5 7B before being applied directly to Mini-Gemini 7B, Parrot 7B, Otter 7B, Vary Toy 1.8B and LLaVA 13B. Evaluation covers the MME hallucination subset, GIFT, AMBER and POPE, with experiments run on two NVIDIA A800 (80GB) GPUs.
GIFT remains limited in scale, and the authors list scaling it up as a key objective for future work; current mitigation methods focus on single-image inputs, leaving multi-image scenarios for later. On the full MME set, models show varying trends across subtasks after adding PLANE, which the authors attribute to differences in architecture and training data.