Few-shot distillation stops feeding only facts: pairing hard samples with counterfactual contexts beats factual-only baselines in most QA settings

Sample choice in few-shot distillation is now a first-class issue: factual samples near the teacher's decision boundary are not good teaching material and measurably hurt the student. Prior few-shot distillation typically drew factual samples under a fixed budget, assuming more samples approach full supervision, without treating distance to the teacher's decision boundary as a selection criterion. Sorting factual samples into three margin bins and distilling with 100 samples from each, the smallest-margin bin gave lower in-domain average EM and F1 than the intermediate and largest-margin bins, a pattern more pronounced on HotpotQA, NewsQA, TriviaQA and NQ than on SQuAD.

The method pairs each hard factual with its counterfactual, using a rewritten context for the same question as extra supervision instead of feeding only the factual answer. Counterfactual augmentation had mostly served classification, while generative open-book QA produces free-form outputs, making counterfactuals hard to define and rarely used for distillation. Counterfactuals are generated by GPT-4o-mini in three stages: identify the answer entity type, generate a same-type alternative answer, then minimally edit the context to support it; candidates are then filtered by a new metric, CFD, which checks both whether the teacher produces the intended alternative answer and whether that answer becomes more predictable given the modified context.

Under the same few-shot budget, paired supervision brings the student closer to the teacher and keeps gains on out-of-domain data. The comparison is factual-only distillation over the same smallest-margin samples and budget, against standard KD, sequence-level KD and layer-wise distillation baselines. Five in-domain plus three out-of-domain benchmarks, T5 and Qwen2.5 families, budgets from 16 to 256 pairs, averaged over five random seeds; under T5 standard KD average F1 improves over the baseline, Qwen2.5 improves as well, and teacher-student agreement EM and F1 exceed standard KD in most settings, though some settings such as HotpotQA with SeqKD do not improve.

Counterfactual generation currently covers only QA samples whose answers are entities, and the method targets open-book QA, where counterfactuals are built by rewriting the supporting context while keeping the question. Experiments span the T5 and Qwen2.5 families, five in-domain and three out-of-domain benchmarks, and few-shot budgets from 16 to 256 pairs, reported as means over five random seeds. Applying this supervision to other generative tasks would require redefining, generating and validating counterfactuals.

The geometric analysis rests on three assumptions: bounded context perturbation, teacher-student agreement on the paired samples, and sufficient counterfactual coverage of the boundary, with answer agreement being an idealized condition. Generative outputs are stochastic, so the teacher may not produce the intended alternative answer, which is why CFD filters candidates before distillation. Performance is sensitive to the CFD threshold: a loose threshold yields negligible gains, moderate thresholds give the largest gains, and gains decline as the threshold becomes more permissive.