Noise-only distillation is no sanitizer: a shared public checkpoint lets a student inherit its teacher's memorized labels and backdoors
Distilling a teacher on pure Gaussian noise still hands the student the teacher's memorized training records and planted backdoors whenever both start from the same public initialization. On MNIST, distilling only never-trained auxiliary outputs yields a membership-inference AUC of 0.86 and a 93% backdoor attack-success lift, while an independently initialized student stays near chance. In a fine-tuning threat model, an attacker with the public base but no data recovers the memorized labels of 93% of private canaries from noise queries alone.