Few-shot distillation stops feeding only factual samples: pairing them with counterfactual contexts beats factual-only baselines across eight QA benchmarks

Few-shot distillation depends on which training samples are chosen, and factual samples near the teacher's decision boundary are the hardest to distill. Prior few-shot distillation used factual samples only, typically the ones with the smallest teacher prediction margins; grouping factual samples into three margin bins of 100 examples each, the smallest-margin bin gave clearly lower EM and F1 than the intermediate and largest-margin bins. The degradation appears across the five in-domain datasets, most pronounced for HotpotQA, NewsQA, TriviaQA and NQ, while SQuAD shows a smaller difference across the three bins.

CofFeQA pairs boundary-near factuals with their counterfactuals, adding supervision around the teacher's decision boundary. Factual-only supervision shows the student what answer the teacher gives for a context, whereas a counterfactual exposes how the teacher changes its answer under a minimal edit to the same question, giving local information on both sides of the boundary. Under T5, averaged over all datasets and 16 to 256 few-shot budgets, F1 rises from 66.42 to 69.27 with standard KD, from 66.37 to 69.28 with LWD, and from 68.63 to 70.52 with SeqKD; under Qwen2.5 standard KD average F1 rises from 66.21 to 67.72.

Counterfactuals are generated by GPT-4o-mini in three stages and then filtered by a new metric, CFD. Generated counterfactuals are not all valid: the teacher may not produce the intended alternative answer, or may infer it without actually relying on the modified context, which weakens the causal relation the student is meant to learn. CFD combines answer agreement with a contextual loss ratio and keeps only counterfactuals where the teacher both produces the intended answer and depends more strongly on the modified context; ablations show the CFD-verified variant gains more in QA performance and teacher-student agreement than the unverified one.

The report covers few-shot distillation for open-book question answering: the five in-domain benchmarks are Natural Questions, SQuAD, NewsQA, TriviaQA and HotpotQA, and the three out-of-domain benchmarks are DROP, DuoRC and RelationExtraction, with T5-Large against T5-Base and Qwen2.5-7B-IT against Qwen2.5-1.5B-IT. Counterfactuals are generated only for samples with entity-based answers, classified as person, location, organization, date or numeric, and samples outside these categories are excluded. The paper lists extending to other answer types and other generative tasks as future directions.

The geometric analysis relies on three idealized conditions: bounded context perturbation, teacher-student agreement on the paired samples, and counterfactuals spread widely enough across the boundary. The paper notes that generative behavior is inherently stochastic, so the teacher may fail to produce the intended alternative answer, and conditioning on the modified context may give only limited predictive benefit, which is why CFD validates counterfactuals before distillation. Performance is sensitive to the CFD threshold: a very strict threshold yields negligible gains, moderate thresholds give the largest gains, and more permissive thresholds reduce the gains gradually.