# Representation Distillation

4 related articles

Subscribe to updates

arXiv Oct 27, 2026
Masked diffusion models lock answers too early: HiLo-RD distills large-window trajectories, lifting GSM8K accuracy by 9.4 points
Masked diffusion language models can fix answer fragments before the supporting reasoning is resolved when the response window is small, and those fragments then become context for later predictions. HiLo-RD uses the same model's large-window trajectory as a frozen teacher and aligns its hidden representations to cropped small-window views at committed positions. At the same small inference budget, GSM8K accuracy rises from 68.84% to 78.24% while mean readable length falls from 362.8 to 271.3 tokens, and the gains extend to code generation without code-domain training.
Masked diffusion language models can fix answer fragments before the supporting reasoning is resolved when the response window is small, and those fragments then become context for later predictions. HiLo-RD uses the same model's large-window trajectory as a frozen teacher and aligns its hidden representations to cropped small-window views at committed positions. At the same small inference budget, GSM8K accuracy rises from 68.84% to 78.24% while mean readable length falls from 362.8 to 271.3 tokens, and the gains extend to code generation without code-domain training.
Zhejiang University Alibaba Group (China)Hong Kong University of Science and Technology Nanyang Technological University The Hong Kong University of Science and Technology (Guangzhou)
Xiang Li（First author）
Representation Distillation Masked Diffusion Language Model Self-Distillation Mathematical Reasoning Budget-Induced Premature Commitment
arXiv Nov 3, 2026
Internal distillation drops the learned projector: RAPID swaps in random projections and cuts up to 21.1% of trainable parameters
Internal distillation aligns a teacher model's intermediate activations with a student's smaller representation space, and existing methods do this by training extra projection matrices that are discarded after training. RAPID replaces them with a fixed random projection built from the Johnson-Lindenstrauss lemma and balances its losses with exponential moving average and homoscedastic uncertainty weighting. In reasoning post-training with Gemma 3 and Qwen 2.5, its four-benchmark averages stay within 0.15 percentage points of learned projection while trainable parameters fall 8.1% in the Qwen experiment and 21.1% when distilling all activations.
Internal distillation aligns a teacher model's intermediate activations with a student's smaller representation space, and existing methods do this by training extra projection matrices that are discarded after training. RAPID replaces them with a fixed random projection built from the Johnson-Lindenstrauss lemma and balances its losses with exponential moving average and homoscedastic uncertainty weighting. In reasoning post-training with Gemma 3 and Qwen 2.5, its four-benchmark averages stay within 0.15 percentage points of learned projection while trainable parameters fall 8.1% in the Qwen experiment and 21.1% when distilling all activations.
University of Southern California
Ryan Swift（First author）
RAPID Representation Distillation Knowledge Distillation Johnson-Lindenstrauss Random Projection Multi-Objective Optimization
researchsquare.com Oct 26, 2026
Behavior classification skips pose training: B-REAL labels raw video frame by frame under 15 ms
A Carnegie Mellon team released B-REAL, a supervised platform that maps raw video directly to behavior labels without pose estimation, with end-to-end inference latency below 15 ms on standard GPU or CPU hardware. On the CalMS21 mouse social-behavior benchmark, one minute of labeled video surpassed A-SOiD and CBAS trained on the full dataset; on fly aggression, it beat both baselines in all eight paired draws from one minute of labels upward. In closed loop, rewarding mice for B-REAL-detected grooming raised the rate of criterion grooming bouts by 129% from early to late sessions.
A Carnegie Mellon team released B-REAL, a supervised platform that maps raw video directly to behavior labels without pose estimation, with end-to-end inference latency below 15 ms on standard GPU or CPU hardware. On the CalMS21 mouse social-behavior benchmark, one minute of labeled video surpassed A-SOiD and CBAS trained on the full dataset; on fly aggression, it beat both baselines in all eight paired draws from one minute of labels upward. In closed loop, rewarding mice for B-REAL-detected grooming raised the rate of criterion grooming bouts by 129% from early to late sessions.
Carnegie Mellon University
Eric A. Yttri（First author）
B-REAL Temporal Shift Module Representation Distillation closed-loop experiment Convolutional Neural Networks
arXiv Oct 1, 2026
RIDE extrapolates RL teachers' hidden-state residuals and matches or beats the teacher on average across four base/teacher pairs
The work proposes RIDE (RL-Induced Direction Extrapolation): on student-generated trajectories it computes the layerwise hidden-state residual between an RL-trained teacher and its pre-RL checkpoint, then regresses the student's hidden states toward targets displaced beyond the teacher along that residual; across four base/RL-teacher pairs (R1-Distill-1.5B, Qwen3-4B, Llama-3.2-3B, Phi-4-mini) RIDE's mean Avg@16 on AIME24, AIME25 and AIMO approaches or exceeds the RL-trained teacher on every pair, the only compared method whose mean does so, and it consistently outperforms output-space extrapolation.
The work proposes RIDE (RL-Induced Direction Extrapolation): on student-generated trajectories it computes the layerwise hidden-state residual between an RL-trained teacher and its pre-RL checkpoint, then regresses the student's hidden states toward targets displaced beyond the teacher along that residual; across four base/RL-teacher pairs (R1-Distill-1.5B, Qwen3-4B, Llama-3.2-3B, Phi-4-mini) RIDE's mean Avg@16 on AIME24, AIME25 and AIMO approaches or exceeds the RL-trained teacher on every pair, the only compared method whose mean does so, and it consistently outperforms output-space extrapolation.
Zhejiang University Rutgers, The State University of New Jersey
Hao Li（First author）
RIDE (RL-Induced Direction Extrapolation)On-Policy Distillation Representation Distillation

You’re all caught up

Masked diffusion models lock answers too early: HiLo-RD distills large-window trajectories, lifting GSM8K accuracy by 9.4 points

Internal distillation drops the learned projector: RAPID swaps in random projections and cuts up to 21.1% of trainable parameters

Behavior classification skips pose training: B-REAL labels raw video frame by frame under 15 ms

RIDE extrapolates RL teachers' hidden-state residuals and matches or beats the teacher on average across four base/teacher pairs