Sangyeon Yoon

Verifiable instruction-following benchmarks check whether output constraints are met, not whether the response answers the request, so frontier models can earn full credit with little substantive content. Using the same verifier as a reinforcement-learning reward produces reward hacking: publicly released OLMo 3 checkpoints show higher constraint satisfaction but lower response quality. CaLIF uses two likelihoods from the frozen reference policy, one under the request and one under the request plus constraints, to define a band that pays a fixed bonus inside and zero above it, raising IFBench constraint satisfaction from 26.98 to 34.64 across five configurations from 1.7B to 8B while holding response quality at 4.59.

Verifiable instruction-following benchmarks check whether output constraints are met, not whether the response answers the request, so frontier models can earn full credit with little substantive content. Using the same verifier as a reinforcement-learning reward produces reward hacking: publicly released OLMo 3 checkpoints show higher constraint satisfaction but lower response quality. CaLIF uses two likelihoods from the frozen reference policy, one under the request and one under the request plus constraints, to define a band that pays a fixed bonus inside and zero above it, raising IFBench constraint satisfaction from 26.98 to 34.64 across five configurations from 1.7B to 8B while holding response quality at 4.59.