Instruction-hierarchy training missed one conflict type: adding goal-goal pairs lifts Qwen3.5-9B conflict resistance by 34.3 points
Synopsis
Existing instruction-hierarchy training sets cover constraint-constraint and constraint-goal conflicts but not two goals that conflict with each other. Decomposing a task into goals and constraints yields 9,544 goal-goal preference pairs built from two text-processing instructions that each process the other's message, and fine-tuning Qwen3.5-9B and Gemma4-12B-it on them with direct preference optimization raises mean conflict resistance by 34.3 and 19.4 points on four instruction-hierarchy benchmarks unseen in training, with small mean utility changes across eight benchmarks.
Interpretation
Instruction-hierarchy training sets were missing an entire conflict type: two goals that each demand a mutually exclusive task. Earlier instruction-hierarchy training sets, including OpenAI's IH-Challenge and NVIDIA's HieraSuite, contain only constraint-constraint and constraint-goal conflicts, with no goal-goal samples. Decomposing a task into goals and constraints and checking both training sets shows the goal-goal type is absent from each.
With those samples added, models hold the higher-priority instruction more often on real-world tasks not seen in training. Fine-tuning on the 9,544 goal-goal pairs alone with direct preference optimization raises mean conflict resistance by 34.3 points for Qwen3.5-9B and 19.4 points for Gemma4-12B-it. Evaluation spans IHEval, RuLES, IH-Benchmark and XIH-Bench, covering rule following, task execution, safety defense and tool use across several message-role boundaries and six languages.
These samples complement existing conflict samples, and combining them beats training on existing samples alone. Adding goal-goal pairs to a subset of HieraSuite raises mean conflict resistance by 18.0 and 10.5 points over training on that subset alone for Qwen3.5-9B and Gemma4-12B-it. The subset holds 6,306 constraint-type preference pairs, giving 15,850 pairs combined; Qwen3.5-9B improves in all ten categories and Gemma4-12B-it in nine of ten.
The conflict-resistance gain does not come at a visible cost to ordinary task performance. On non-conflicting samples from eight benchmarks, training on goal-goal pairs alone and adding them to existing conflict training both give mean utility similar to the corresponding original models. The eight benchmarks are MMLU-Pro, GPQA Diamond, IFEval, GSM8K, IHEval, RuLES, IH-Benchmark and XIH-Bench.
Perspective
The findings rest on two models, Qwen3.5-9B and Gemma4-12B-it, four instruction-hierarchy benchmarks and eight utility benchmarks, with training samples drawn from English system-message and user-message text-processing instructions.
Evaluation uses fixed benchmark protocols without attacks adaptively optimized against the fine-tuned models, so whether the gains persist under such attacks is unknown; a small fraction of cases shows over-defense with conflict resistance above 100%, indicating that conflict-only training is not sufficient and no-conflict samples need to be mixed in.