Multimodal quantization stops text from drowning out vision: one shared transform reconciles both gradients, 66.51 average on the 32B model at W4A6
Synopsis
When a multimodal large language model is quantized to low bit-widths, text and vision tokens must learn one shared channel-wise transform, and their gradients can point in opposite directions or differ wildly in magnitude. MCTQ splits the reconstruction objective by modality, then combines the two gradients into a single update direction through a minimum-norm convex combination, keeping one shared transform and adding no inference overhead. On Qwen2.5-VL-7B at W4A6 it reaches 66.26 average accuracy, 1.30 points above OmniQuant; Qwen2.5-VL-32B reaches 66.51 in the same setting.
Interpretation
Post-training quantization for multimodal large language models has an extra step: text tokens and vision tokens share one channel-wise transform, and the two token types do not want the same channel scaling. Earlier channel-wise quantization methods either fixed the scaling from activation statistics or optimized a single mixed reconstruction loss, summing the two modalities' gradients before checking whether they agree. Computing text and vision reconstruction gradients separately with respect to the shared transform parameters puts many samples in the negative-cosine region or at large gradient-norm ratios; layer-wise histograms show the pattern across layers and transform types.
The approach splits one mixed objective into separate text and vision reconstruction objectives, then merges the two gradients into one update direction at the gradient level. The merge takes the minimum-norm convex combination of the two gradients, with a closed-form weight clipped to the interval from 0 to 1, so conflicting directions yield a compromise and a large magnitude gap automatically down-weights the dominant side. On Qwen2.5-VL-7B at W4A6, rescaling only the text-side gradient to a median text-to-vision norm ratio of 0.85 gives the best average accuracy, 70.76; shifting toward either modality lowers it.
The pipeline still learns a single shared transform and folds it into the quantized weights, adding no modality-specific parameters or branches at inference. By contrast, giving each modality its own transform avoids the conflict but costs extra parameters and inference overhead. Measured on a single A100-80G at W4A8, prefill length 1024 with batch size 1 gives a 1.31x speedup over FP16 and peak memory of 10.69 GB; batch size 4 at length 2048 gives a 1.33x speedup and 1.10x memory saving.
Perspective
Validation covers three vision-language models, Qwen2.5-VL-7B, Qwen2.5-VL-32B and LLaVA-OneVision-1.5-8B, with W4A6 as the primary setting and additional W4A8 results plus W8A8 on Qwen2.5-VL-7B; evaluation uses six benchmarks, OCRBench, SEEDBench, MMMU, VizWiz, ScienceQA and TextVQA, with 128 image-caption pairs from COCO val2017 as calibration data. Efficiency figures are measured on a single A100-80G at W4A8.
Under the W4A4 stress test MCTQ raises average accuracy from the best baseline's 6.30 to 19.40, but a large gap to the FP16 baseline's 68.00 remains, and the authors treat this setting as a stress test rather than deployment-ready. The method currently handles only text and vision, and extending to omni-modal models such as speech may bring higher-order gradient conflicts. Reconstruction epochs rising from 30 to 50 show mild overfitting, and average accuracy varies within 0.9 points across calibration sets of 16 to 512 samples.