# BridgeKV

BridgeKV lets a large model in a model family run prefill once, then a lightweight KV-stitching module translates the large model's prompt cache into the small model's native cache layout, so every output token is generated by the small model alone. Across six model pairs and five datasets, it scores above fine-tuned small-model baselines with matched trainable parameter budgets in all 30 settings, closing up to 80% of the quality gap to running the large model for both phases. On one A40, Llama-3.1-Instruct-8B→Llama-3.2-Instruct-1B serves a 1K-token prompt with 256 output tokens in 2.96 s, close to the 2.80 s of the small model alone and 3.0× faster than the 8.83 s of the large model alone.

BridgeKV lets a large model in a model family run prefill once, then a lightweight KV-stitching module translates the large model's prompt cache into the small model's native cache layout, so every output token is generated by the small model alone. Across six model pairs and five datasets, it scores above fine-tuned small-model baselines with matched trainable parameter budgets in all 30 settings, closing up to 80% of the quality gap to running the large model for both phases. On one A40, Llama-3.1-Instruct-8B→Llama-3.2-Instruct-1B serves a 1K-token prompt with 256 output tokens in 2.96 s, close to the 2.80 s of the small model alone and 3.0× faster than the 8.83 s of the large model alone.