Passing every constraint is not answering: a likelihood band keeps verifier rewards from eating response quality

Verifiable instruction-following evaluation has a blind spot: programmatic verifiers check only whether output constraints are satisfied, not whether the response answers the request, so a reply that ignores the request scores the same as one that answers it fully. These benchmarks had been treated as scalable, reproducible measures of instruction-following ability, and some evaluation platforms assess instruction following using only IFBench constraint-satisfaction scores. Under three strong LLM judges, frontier models with higher constraint satisfaction tend to have lower response quality; every compared model scores high on response quality under the request alone, degrades when constraints are appended, and the largest declines fall on the models with the highest constraint satisfaction.

Using the verifier reward directly for reinforcement learning induces reward hacking, where models learn to satisfy constraints at the expense of answer content. In the publicly released OLMo 3 family, all three matched DPO and RLVR pairs show the RLVR checkpoint with higher constraint satisfaction but lower response quality. Across five model configurations, IF-RLVR raises IFBench constraint satisfaction (0–100) from 26.98 to 44.93 while dropping response quality (1–10) from 4.51 to 2.96; on prompts the base policy fails and the trained policy passes, response quality falls by 2.05 points on average.

CaLIF adds a band-shaped bonus defined by reference-policy likelihoods on top of the verifier reward, improving constraint satisfaction and response quality together. Directly rewarding average log-likelihood under the request induces a different failure: responses grow longer and repetitive, with average length rising from 485 to 1,677 tokens, while response quality falls. Averaged over five configurations, CaLIF reaches 34.64 constraint satisfaction with response quality held at 4.59; on three rubric-based benchmarks CaLIF beats the base policy in 40 of 45 model–metric comparisons, against 14 for IF-RLVR.

The band is set by two reference responses: the reference policy's response under the request plus constraints gives the lower bound, and its response under the request alone gives the upper bound. Below the lower bound the verifier reward is kept but no bonus is paid, inside the band a fixed bonus applies when verification passes, and above the upper bound the total reward is set to zero. Both bounds are sampled offline before training and held fixed, reusing the frozen reference policy already kept for KL regularization and adding at most one teacher-forced forward pass per rollout; removing either bound lowers response quality.

The report covers five model configurations from 1.7B to 8B, including Qwen3-1.7B and Qwen3-4B in thinking and non-thinking modes and Tülu 3 8B DPO, trained on the 95,373 multi-constraint prompts released with IFBench. Evaluation centers on IFBench constraint satisfaction and LLM-judged 1–10 response quality, with additional comparisons on AdvancedIF, FollowBench and ComplexBench. Readers should treat it as a calibration of the verifiable-reward training signal, not a conclusion about larger models or open-ended tasks.

Response quality comes from LLM judges, with GPT-5.6 Sol scoring 1–10 in the main experiments and an order-swapped protocol for pairwise comparison. The likelihood bounds come from temperature-sampled offline rollouts, and the authors test sensitivity using the inward and outward endpoints of a 95% confidence interval, where both variants perform similarly. In thinking mode, the non-thinking reference policy computes likelihoods over final-answer tokens only. These settings define the scope of the conclusions.