Ryan Swift

Internal distillation aligns a teacher model's intermediate activations with a student's smaller representation space, and existing methods do this by training extra projection matrices that are discarded after training. RAPID replaces them with a fixed random projection built from the Johnson-Lindenstrauss lemma and balances its losses with exponential moving average and homoscedastic uncertainty weighting. In reasoning post-training with Gemma 3 and Qwen 2.5, its four-benchmark averages stay within 0.15 percentage points of learned projection while trainable parameters fall 8.1% in the Qwen experiment and 21.1% when distilling all activations.

Internal distillation aligns a teacher model's intermediate activations with a student's smaller representation space, and existing methods do this by training extra projection matrices that are discarded after training. RAPID replaces them with a fixed random projection built from the Johnson-Lindenstrauss lemma and balances its losses with exponential moving average and homoscedastic uncertainty weighting. In reasoning post-training with Gemma 3 and Qwen 2.5, its four-benchmark averages stay within 0.15 percentage points of learned projection while trainable parameters fall 8.1% in the Qwen experiment and 21.1% when distilling all activations.