# Singular Value Decomposition

Reasoning vision-language models must hold both multimodal context and a growing reasoning trace in their KV cache, and existing low-rank compression calibrates on static inputs, so its divergence from the dense model grows with generation length. TraceKV instead calibrates on reasoning traces the dense model generates itself, weights each KV state by its sensitivity to policy divergence before the SVD, and allocates ranks globally. Across four models including Qwen2.5-VL-7B, a 70% rank ratio with 4-bit quantization gives 73.41% overall accuracy versus 73.58% for the dense model, with the KV cache reduced 5.64 times.

Reasoning vision-language models must hold both multimodal context and a growing reasoning trace in their KV cache, and existing low-rank compression calibrates on static inputs, so its divergence from the dense model grows with generation length. TraceKV instead calibrates on reasoning traces the dense model generates itself, weights each KV state by its sensitivity to policy divergence before the SVD, and allocates ranks globally. Across four models including Qwen2.5-VL-7B, a 70% rank ratio with 4-bit quantization gives 73.41% overall accuracy versus 73.58% for the dense model, with the KV cache reduced 5.64 times.

Isolating the reinforcement-learning-stage parameter update of a language model, keeping only the single dominant direction of each matrix rescaled to the original magnitude, and injecting it into the language modules of a vision-language model improves matched full-update interpolation in 12 of 15 backbone-benchmark comparisons across three model families and five visual-reasoning benchmarks, including an 8.55 percentage-point MathVision gain on the Qwen recipient. Controls show that matching update magnitude or using arbitrary low-rank directions does not reproduce these gains, indicating that what is acquired during post-training and what remains transferable across models are not the same thing.

Isolating the reinforcement-learning-stage parameter update of a language model, keeping only the single dominant direction of each matrix rescaled to the original magnitude, and injecting it into the language modules of a vision-language model improves matched full-update interpolation in 12 of 15 backbone-benchmark comparisons across three model families and five visual-reasoning benchmarks, including an 8.55 percentage-point MathVision gain on the Qwen recipient. Controls show that matching update magnitude or using arbitrary low-rank directions does not reproduce these gains, indicating that what is acquired during post-training and what remains transferable across models are not the same thing.

The work defines each singular value's paired left and right singular vectors as a singular-vector channel, estimates each channel's adaptation benefit from current-task gradients and its forgetting cost from activation second moments on an unlabeled general-domain corpus, then selects trainable channels via knee-based cost screening, Pareto-front filtering, and Otsu thresholding; across four LLM families, eight TRACE tasks, and three task orders, SVC preserves pretrained general capabilities while achieving strong downstream continual-learning performance.

The work defines each singular value's paired left and right singular vectors as a singular-vector channel, estimates each channel's adaptation benefit from current-task gradients and its forgetting cost from activation second moments on an unlabeled general-domain corpus, then selects trainable channels via knee-based cost screening, Pareto-front filtering, and Otsu thresholding; across four LLM families, eight TRACE tasks, and three task orders, SVC preserves pretrained general capabilities while achieving strong downstream continual-learning performance.

The authors introduce Hierarchical Spectral Shrinkage (HSS), an empirical-Bayes framework that shrinks each task's leading spectral directions toward a data-adaptively learned common basis with shrinkage varying across tasks and spectral components, yielding the posterior mean of singular vectors in a surrogate Bayesian regression; across synthetic experiments and three immune-cell gene-expression datasets (sample sizes 156, 628, 146), HSS improves estimation accuracy and out-of-sample performance over separate, fully pooled, and shared-subspace approaches.

The authors introduce Hierarchical Spectral Shrinkage (HSS), an empirical-Bayes framework that shrinks each task's leading spectral directions toward a data-adaptively learned common basis with shrinkage varying across tasks and spectral components, yielding the posterior mean of singular vectors in a surrogate Bayesian regression; across synthetic experiments and three immune-cell gene-expression datasets (sample sizes 156, 628, 146), HSS improves estimation accuracy and out-of-sample performance over separate, fully pooled, and shared-subspace approaches.