CRD distills reasoning via multi-teacher cross-feedback and step-level stitching, reaching 97.3% on MATH-500 and 70.3% on AIME'25 with a 4B student trained on only 50K examples
The work proposes Collaborative Reasoning Distillation (CRD), in which DeepSeek-R1 and Gemini 2.5 Flash Thinking iteratively critique each other's reasoning, a dual-judge system scores each step's logical validity independently of the final answer, and a Viterbi-style coherence-aware algorithm stitches high-quality steps across teachers; students are trained with Reasoning Quality Optimization under curriculum budget constraints, and CRD-4B reaches 97.3% on MATH-500 and 70.3% on AIME'25 using only 50K training examples.