TraceKV shrinks reasoning-VLM KV cache about sixfold: on-policy calibration keeps reasoning accuracy near the dense model
Synopsis
Reasoning vision-language models must hold both multimodal context and a growing reasoning trace in their KV cache, and existing low-rank compression calibrates on static inputs, so its divergence from the dense model grows with generation length. TraceKV instead calibrates on reasoning traces the dense model generates itself, weights each KV state by its sensitivity to policy divergence before the SVD, and allocates ranks globally. Across four models including Qwen2.5-VL-7B, a 70% rank ratio with 4-bit quantization gives 73.41% overall accuracy versus 73.58% for the dense model, with the KV cache reduced 5.64 times.
Interpretation
Low-rank KV compression can be nearly lossless on reasoning vision-language models: at a 70% rank ratio overall accuracy is 73.33% against 73.58% for the dense model. Earlier low-rank methods calibrate on static multimodal inputs and work on short-answer tasks, but on generated reasoning states the next-token policy-KL divergence from the dense model grows sharply with generation length, with logical errors and infinite loops appearing. On ScienceQA, MathVista and MMMU, at a 50% rank ratio TraceKV reaches 70.28% overall accuracy, 7.09 percentage points above the best baseline, Palu.
Compression no longer targets KV reconstruction but preservation of the dense model's predictive policy. Conventional objectives minimize KV reconstruction error over calibration activations and treat all KV states alike. Calibration uses 256 samples from the ScienceQA train split, with each model greedily generating its own reasoning traces up to 512 tokens; on Qwen2.5-VL-7B-Instruct, adding ground-truth solution traces lowers reasoning-state reconstruction error from 10.27 to 9.67, and on-policy traces lower it further to 8.88.
Sensitivity-weighted SVD and global rank allocation concentrate the limited rank budget on the KV states and singular components that most affect the policy. The reconstruction objective weights all KV states equally, yet equal-magnitude perturbations differ greatly in effect: many visual states are insensitive while some textual and reasoning states are highly sensitive. In the component ablation, ground-truth-trace calibration alone raises average accuracy from 58.41% to 60.61%, on-policy calibration to 62.09%, sensitivity-aware SVD to 63.68%, and adding global rank allocation reaches 66.57%, 8.16 percentage points above the baseline.
Perspective
The demonstrated scope is four reasoning vision-language models (Qwen2.5-VL-7B-Instruct, InternVL3-8B, Qwen3-VL-8B/32B-Thinking) evaluated on three visual reasoning benchmarks, ScienceQA, MathVista and MMMU, with calibration from 256 ScienceQA train-split samples; decoding latency is measured on Qwen2.5-VL-7B-Instruct at batch size 16 and context lengths from 4K to 32K.
The compression objective minimizes policy-KL over the dense model's reasoning traces, while the compressed model meets its own autoregressive states at inference, and this distribution mismatch may accumulate errors over long reasoning trajectories; calibration still relies on a limited set of task-specific examples and their model-generated traces, and whether the strategy generalizes to broader domains or long-form generation beyond reasoning remains open.