# Vision-Language Model

Multimodal Rotary Position Embedding (MRoPE), widely used in vision-language models, assigns height and width coordinates to text tokens that have no inherent spatial structure, adding positional preferences to text-vision attention. The study separates intra-visual spatial encoding from cross-modal spatial effects, probing official pretrained checkpoints without training and running controlled training at 2B and 4B from identical initial parameters. Cross-modal spatial effects provide little extra benefit as spatial references and can even impair performance, while intra-visual spatial encoding remains useful for fine-grained grounding.

Multimodal Rotary Position Embedding (MRoPE), widely used in vision-language models, assigns height and width coordinates to text tokens that have no inherent spatial structure, adding positional preferences to text-vision attention. The study separates intra-visual spatial encoding from cross-modal spatial effects, probing official pretrained checkpoints without training and running controlled training at 2B and 4B from identical initial parameters. Cross-modal spatial effects provide little extra benefit as spatial references and can even impair performance, while intra-visual spatial encoding remains useful for fine-grained grounding.

Ultra-high-resolution remote sensing question answering has assumed that once the right view is chosen, a vision-language model's counts and directions can be trusted. GeoScope treats the frozen model as a sensor to be calibrated, measures the number of objects it counts reliably in one view (about five for most models on DOTA aerial images), splits only views beyond that horizon, and computes directions explicitly from grounded boxes. Without training it raises Qwen3-VL-8B by 16.3 and 10.5 points on XLRS-Bench regional counting and spatial relations, and by 16.8 and 36.2 points on UHR-Micro global and regional counting.

Ultra-high-resolution remote sensing question answering has assumed that once the right view is chosen, a vision-language model's counts and directions can be trusted. GeoScope treats the frozen model as a sensor to be calibrated, measures the number of objects it counts reliably in one view (about five for most models on DOTA aerial images), splits only views beyond that horizon, and computes directions explicitly from grounded boxes. Without training it raises Qwen3-VL-8B by 16.3 and 10.5 points on XLRS-Bench regional counting and spatial relations, and by 16.8 and 36.2 points on UHR-Micro global and regional counting.

A robot that files a hallway defibrillator under fire extinguisher would carry that wrong description in memory. MIT researchers added uncertainty quantification to the existing DAAAM memory system, so the robot compares descriptions of the same object from different viewpoints, estimates how unsure it is about what it saw, and selects extra views to fuse a more accurate caption. Against baseline methods, the system raised object-description accuracy by 5 percent and resolved ambiguity for about 92 percent of objects using only two additional captions per object.

A robot that files a hallway defibrillator under fire extinguisher would carry that wrong description in memory. MIT researchers added uncertainty quantification to the existing DAAAM memory system, so the robot compares descriptions of the same object from different viewpoints, estimates how unsure it is about what it saw, and selects extra views to fuse a more accurate caption. Against baseline methods, the system raised object-description accuracy by 5 percent and resolved ambiguity for about 92 percent of objects using only two additional captions per object.

Large vision-language models process hundreds to thousands of visual tokens per image, and inference cost rises with them. Earlier training-free pruning methods ranked all visual tokens by one importance criterion, overlooking that tokens take on different roles as encoding proceeds. BtsPruner locates the role transition from changes in attention concentration, then picks globally referenced representative tokens from later layers and recursively retrieves local detail tokens from earlier layers, preserving both roles under a fixed budget; on LLaVA-1.5-7B it keeps only 32 of 576 visual tokens (5.6%) and retains 91.6% of the dense model's performance.

Large vision-language models process hundreds to thousands of visual tokens per image, and inference cost rises with them. Earlier training-free pruning methods ranked all visual tokens by one importance criterion, overlooking that tokens take on different roles as encoding proceeds. BtsPruner locates the role transition from changes in attention concentration, then picks globally referenced representative tokens from later layers and recursively retrieves local detail tokens from earlier layers, preserving both roles under a fixed budget; on LLaVA-1.5-7B it keeps only 32 of 576 visual tokens (5.6%) and retains 91.6% of the dense model's performance.

Aleph Alpha distilled its self-checking agentic parsing harness into a single vision-language model, Parsy, which handles each page in one pass. The harness reached 86.0 on ParseBench by running several models and up to three passes per page, too slow and costly for millions of pages a month. Parsy was trained on harness-labeled public documents whose labels were cleaned with an independent layout model, an ink-coverage check and a second OCR model; it scores 87.9 on ParseBench, parses hundreds of thousands of pages per day on one H100, and costs one to two orders of magnitude less per page.

Aleph Alpha distilled its self-checking agentic parsing harness into a single vision-language model, Parsy, which handles each page in one pass. The harness reached 86.0 on ParseBench by running several models and up to three passes per page, too slow and costly for millions of pages a month. Parsy was trained on harness-labeled public documents whose labels were cleaned with an independent layout model, an ink-coverage check and a second OCR model; it scores 87.9 on ParseBench, parses hundreds of thousands of pages per day on one H100, and costs one to two orders of magnitude less per page.

Vision-language models often describe objects that are not in the image, and earlier fixes intervened at the output level or across whole attention layers, which cannot separate heads that support faithful captions from heads that drive hallucination. HACP scores each attention head at inference time by the spectral change of its perturbed representation, then suppresses hallucination-prone heads and reinforces faithful ones during decoding without training. On a 1000-image multi-label COCO subset it lowers CHAIR-s/i from 66.7/18.0 to 43.4/8.6 and raises F1 from 73.0 to 77.0.

Vision-language models often describe objects that are not in the image, and earlier fixes intervened at the output level or across whole attention layers, which cannot separate heads that support faithful captions from heads that drive hallucination. HACP scores each attention head at inference time by the spectral change of its perturbed representation, then suppresses hallucination-prone heads and reinforces faithful ones during decoding without training. On a 1000-image multi-label COCO subset it lowers CHAIR-s/i from 66.7/18.0 to 43.4/8.6 and raises F1 from 73.0 to 77.0.

Hallucinations in large vision-language models partly trace to CLIP-style encoders that favor main objects and skip image detail. PLANE attaches three LoRA visual experts to CLIP, each trained on object, attribute or relation captions, and an instruction-aware gate fuses the four image representations at token level. Trained once on LLaVA-1.5 7B, it plugs into other models sharing the same encoder; on LLaVA-1.5 7B the MME hallucination subset total rises from 555.38 to 582.34 and the GIFT hallucination rate falls from 23.39% to 20.30%.

Hallucinations in large vision-language models partly trace to CLIP-style encoders that favor main objects and skip image detail. PLANE attaches three LoRA visual experts to CLIP, each trained on object, attribute or relation captions, and an instruction-aware gate fuses the four image representations at token level. Trained once on LLaVA-1.5 7B, it plugs into other models sharing the same encoder; on LLaVA-1.5 7B the MME hallucination subset total rises from 555.38 to 582.34 and the GIFT hallucination rate falls from 23.39% to 20.30%.

Instead of drawing on a fixed command library, a quadruped exploring unknown space hands the user's high-level intention to a large language model that writes an executable program restricted to a robot function library, while a vision-language model supplies open-ended semantic labels and geometric and semantic thresholds trigger replanning. In Gazebo simulation, Qwen3.5 and Kimi-K2.5 reach about 82% overall success and GPT-5-nano about 46%; finite tasks hold up well, while open-ended tasks vary more.

Instead of drawing on a fixed command library, a quadruped exploring unknown space hands the user's high-level intention to a large language model that writes an executable program restricted to a robot function library, while a vision-language model supplies open-ended semantic labels and geometric and semantic thresholds trigger replanning. In Gazebo simulation, Qwen3.5 and Kimi-K2.5 reach about 82% overall success and GPT-5-nano about 46%; finite tasks hold up well, while open-ended tasks vary more.

Across five vision-centric tasks, adding labeled demonstrations to the context sharply raises accuracy on counterfactual examples that conflict with training priors while leaving prior-aligned accuracy nearly unchanged, indicating the models already have the visual capability and were being overridden by learned regularities.

Across five vision-centric tasks, adding labeled demonstrations to the context sharply raises accuracy on counterfactual examples that conflict with training priors while leaving prior-aligned accuracy nearly unchanged, indicating the models already have the visual capability and were being overridden by learned regularities.

SP-DocReader is a self-play framework for document OCR that aligns reference and generated readings with a longest common subsequence, applies contrastive relative scoring and direct negative log-likelihood supervision only at unmatched positions, and trains only the OCR module while the visual encoder and language backbone stay frozen; across Qwen3-VL-4B and InternVL-3.5-4B, three self-play rounds reduce Vary-600K character error rate relative to two-epoch supervised fine-tuning and improve DocBank, IIT-CDIP, DocVQA and InfographicVQA scores.

SP-DocReader is a self-play framework for document OCR that aligns reference and generated readings with a longest common subsequence, applies contrastive relative scoring and direct negative log-likelihood supervision only at unmatched positions, and trains only the OCR module while the visual encoder and language backbone stay frozen; across Qwen3-VL-4B and InternVL-3.5-4B, three self-play rounds reduce Vary-600K character error rate relative to two-epoch supervised fine-tuning and improve DocBank, IIT-CDIP, DocVQA and InfographicVQA scores.