Constraint violations stop counting the same: severity-weighted preference signals let an 8B model match multi-agent planners in one pass

On planning tasks with hard constraints, a model no longer learns from a small budget overshoot and a severe one as if they were the same mistake. Direct preference optimization previously assigned the same implicit reward gap to every rejected sample, distinguishing neither violation severity nor hard constraints from soft preferences. On TravelPlanner, the 8B model trained with this objective reaches an 89.2% pass rate, close to the 89.4% of a multi-agent system, at 2.1 seconds of single-pass latency versus 28.5 seconds.

The training signal comes from a deterministic symbolic verifier rather than a learned reward model. The verifier divides each raw violation amount by that constraint type's median severity and caps the result at 1.0, yielding a continuous severity that a bounded transform turns into the margin inside the preference loss. In a margin-source comparison, verifier-normalized severity reaches 89.4% pass rate, against 65.6% for a random margin, 67.8% for a length-difference margin, and 73.3% for a reward-model gap.

Hard constraints and soft preferences are split into two levels, and soft preferences are optimized only once hard constraints hold. Earlier methods mixed the two categories in one preference signal, which could trade a hard constraint against preference satisfaction. Among TravelPlanner failures, no error appears in which a soft preference is incorrectly prioritized over a hard constraint, consistent with the intended behavior of the lexicographic objective.

Preference pairs are synthesized from procedural constraint profiles and minimal-edit distillation to reduce style and length bias. Full-rewrite distillation changes large amounts of constraint-irrelevant text, and standard preference optimization exploits those stylistic differences as an implicit signal. Minimal edits average about 32 tokens against 245 for full rewrites; under standard preference optimization, switching to minimal edits raises pass rate by 14.2 points, and by 16.6 points with the margin objective.

The pipeline synthesizes and verifies preference pairs in two domains, travel planning and retail recommendation, and evaluates on 180 TravelPlanner queries, 200 NaturalPlan meeting-scheduling instances, and out-of-distribution PlanBench Blocksworld problems. Training runs on four 80GB A100 GPUs, with supervised fine-tuning taking about 4 hours, preference optimization about 2.5 hours, and data synthesis costing about $320. The authors position the method for settings where constraints can be checked deterministically, and state that they have not tested whether the gains transfer to substantially smaller or larger model scales.

The method depends on a symbolic verifier, so extending it to open-world environments where constraints cannot be checked deterministically remains an open problem. Budget violations account for the largest share of TravelPlanner failures at 36.8%, which the authors read as implicit arithmetic remaining a bottleneck for longer plans. The authors also report that preliminary experiments with Qwen3-8B in thinking mode further reduce budget violations, but exclude them from the main results to keep thinking and non-thinking inference regimes comparable, and note that teacher-guided synthesis is costly, so larger-scale deployment may require more efficient self-play strategies.