Few-shot distillation stops feeding only facts: pairing hard samples with counterfactual contexts beats factual-only baselines in most QA settings
Few-shot distillation performance depends heavily on which training samples are chosen, and factual samples near the teacher's decision boundary can actually degrade it. CofFeQA pairs those hard factuals with counterfactuals, minimally edited contexts that make the teacher switch to an alternative answer, adding supervision around the boundary. Across five in-domain and three out-of-domain open-book QA benchmarks with T5 and Qwen2.5, it beats factual-only distillation under the same few-shot budget in most settings, though not in every one.