# Cross-Layer KV Sharing

DepthGraft turns cross-layer KV sharing from a cache-saving inference trick into a structural regularizer during training: deep layers no longer compute their own KV but rebuild it from a stride-aligned shallow–middle source pair using learnable channel-wise coefficients. Gradients from deep layers then flow back along each sharing edge into the source KV projections, sharpening key retrieval directions while enriching the value content subspace. Across dense and MoE models from 0.5B to 16B parameters, downstream averages improve consistently, with a 7.0-point absolute gain on MMLU-Pro for the 16B MoE and roughly one third less KV cache.

DepthGraft turns cross-layer KV sharing from a cache-saving inference trick into a structural regularizer during training: deep layers no longer compute their own KV but rebuild it from a stride-aligned shallow–middle source pair using learnable channel-wise coefficients. Gradients from deep layers then flow back along each sharing edge into the source KV projections, sharpening key retrieval directions while enriching the value content subspace. Across dense and MoE models from 0.5B to 16B parameters, downstream averages improve consistently, with a 7.0-point absolute gain on MMLU-Pro for the 16B MoE and roughly one third less KV cache.

The work introduces WhiteMatter, which uses a learned router to mix past-token representations from any depth into shared key-value (KV) channels so every layer can draw on cross-layer information, together with cyclic Gauss–Seidel iteration to speed up training and prefill; under matched training-token budgets, half-cache WhiteMatter improves perplexity and average downstream scores over matched standard Transformers at two scales up to about 1.3B parameters, while the full-cache version performs comparably to a deeper standard Transformer.

The work introduces WhiteMatter, which uses a learned router to mix past-token representations from any depth into shared key-value (KV) channels so every layer can draw on cross-layer information, together with cyclic Gauss–Seidel iteration to speed up training and prefill; under matched training-token budgets, half-cache WhiteMatter improves perplexity and average downstream scores over matched standard Transformers at two scales up to about 1.3B parameters, while the full-cache version performs comparably to a deeper standard Transformer.