Internal distillation drops the learned projector: RAPID swaps in random projections and cuts up to 21.1% of trainable parameters
Synopsis
Internal distillation aligns a teacher model's intermediate activations with a student's smaller representation space, and existing methods do this by training extra projection matrices that are discarded after training. RAPID replaces them with a fixed random projection built from the Johnson-Lindenstrauss lemma and balances its losses with exponential moving average and homoscedastic uncertainty weighting. In reasoning post-training with Gemma 3 and Qwen 2.5, its four-benchmark averages stay within 0.15 percentage points of learned projection while trainable parameters fall 8.1% in the Qwen experiment and 21.1% when distilling all activations.
Interpretation
Internal distillation no longer needs a trained projection matrix to bridge the dimension gap between teacher and student representations; a fixed random projection compresses teacher activations directly into the student's dimension. Earlier internal-distillation methods learned one projection per distilled activation, mapping student activations upward into the teacher's space, and those parameters were thrown away once training ended. On two in-family distillation pairs, Gemma 3 4B into Gemma 3 270M and Qwen 2.5 3B into Qwen 2.5 0.5B, RAPID and learned projection differ by no more than 0.15 percentage points in overall average across ARC-C, BBH, GPQA and MMLU; the largest single delta is +1.05 percentage points for RAPID on BBH with SmolTalk.
With the projection fixed, the whole job of closing the representation gap falls to the student, while the teacher side supplies only a target that does not move during training. When a learned projector trains jointly with the student, the projector can change both the residual and how that residual reaches the student, so part of the alignment may sit in a projector discarded after training. Experiments run post-training on 40M tokens of reasoning data at sequence length 1024, using a reasoning-curated version of SmolTalk and unmodified OpenThoughts-114k, with three random seeds per model averaged.
A multi-objective loss combines language modeling, output-distribution divergence and internal-activation matching under dynamic weights set jointly by learnable homoscedastic uncertainty and an exponential-moving-average progress ratio. The three internal-distillation losses differ widely in magnitude and carry noisy signals, so fixed equal weighting cannot shift emphasis across training stages. Weights are normalized across active objectives, clamped within bounds, and computed without gradient tracking; distilling all activations uses 18 layers times 3 activations plus one embedding projector, 55 projectors in total.
Perspective
The demonstrated scope is in-family distillation within the Gemma 3 and Qwen 2.5 families, limited to reasoning-focused supervised fine-tuning post-training at sequence length 1024 on 40M tokens; all experiments ran on a single server with two RTX A6000 48GB GPUs.
The authors state that compute constrained model size, sequence length and teacher fine-tuning, so full R1-style long-context reasoning distillation and evaluation were not done; RAPID is also evaluated only in reasoning-focused supervised fine-tuning, not in pretraining, mid-training, general instruction tuning, tool-use training, or as an initialization for later reinforcement learning and other post-training stages. When distilling all activations, RAPID's average on Qwen 2.5 is 2.2 percentage points below learned projection; the same passage also calls the largest gap in all experiments a 1.67 percentage-point lead for learned projection, and the two figures disagree, so both are reported as stated.