# DepthGraft

DepthGraft turns cross-layer KV sharing from a cache-saving inference trick into a structural regularizer during training: deep layers no longer compute their own KV but rebuild it from a stride-aligned shallow–middle source pair using learnable channel-wise coefficients. Gradients from deep layers then flow back along each sharing edge into the source KV projections, sharpening key retrieval directions while enriching the value content subspace. Across dense and MoE models from 0.5B to 16B parameters, downstream averages improve consistently, with a 7.0-point absolute gain on MMLU-Pro for the 16B MoE and roughly one third less KV cache.

DepthGraft turns cross-layer KV sharing from a cache-saving inference trick into a structural regularizer during training: deep layers no longer compute their own KV but rebuild it from a stride-aligned shallow–middle source pair using learnable channel-wise coefficients. Gradients from deep layers then flow back along each sharing edge into the source KV projections, sharpening key retrieval directions while enriching the value content subspace. Across dense and MoE models from 0.5B to 16B parameters, downstream averages improve consistently, with a 7.0-point absolute gain on MMLU-Pro for the 16B MoE and roughly one third less KV cache.