Constrained decoding is not the reasoning killer: with matched prompts, validity constraints gain up to 17.4%
Synopsis
Constrained decoding is often assumed to degrade LLM reasoning, but once the prompt and expected output format are aligned, causal ablations show the apparent gap comes mainly from prompt-level differences rather than decoder masking. The work splits constraints into format, validity and reasoning types and tests them across four model families and eleven open-source models; validity constraints, which remove invalid answers, give the most consistent gains, up to 17.4%, while reasoning constraints help only when the enforced trajectory is itself beneficial.
Interpretation
When the prompt and expected output format stay fixed, constrained decoding does not weaken LLM generation and instead matches or beats unconstrained decoding. Earlier observations attributed accuracy drops under structured formats to constrained decoding itself, motivating fixes such as adding a reason field or two-step generate-then-format pipelines. Causal ablations across four model families and eleven open-source models, using several state-of-the-art constrained decoding strategies, show the apparent degradation stems largely from non-equivalent prompts and output formats rather than decoder-level masking.
Constraints fall into format, validity and reasoning types, each removing a different part of the output space with a different effect. Format constraints only enforce output structure, validity constraints mask invalid continuations by requirements such as data type or length, and reasoning constraints restrict the reasoning trajectory. On GSM8K, Last Letter Concatenation, AIME 2026 and LastLetter-Extended, which extends the name count from four to eight, validity constraints give the most consistent gains, up to 17.4%.
Reasoning constraints can improve reasoning and accuracy, but only when the enforced trajectory is itself beneficial. On GSM8K and AIME 2026, forcing the model to begin with “Let’s think step by step” generally improves performance, yet Qwen3.5-4B drops 6.6% on AIME 2026. That contrast shows reasoning-constraint gains depend on model and task, and identifying beneficial traces often requires prior knowledge that is unavailable in practice.
Perspective
The conclusions rest on eleven open-source models from four model families and on benchmarks including GSM8K, Last Letter, LastLetter-Extended and AIME 2026, with relatively simple JSON formats; validity constraints are evaluated mainly on LastLetter and LastLetter-Extended, plus a preliminary content-filtering experiment on IMDb.
Reasoning-constraint effects vary with model and task, with Qwen3.5-4B dropping 6.6% on AIME 2026; identifying beneficial reasoning traces requires prior information, and the experiments demonstrate feasibility using patterns from prior work rather than offering a way to find them. Constrained decoding also requires determining allowed tokens and masking at each step, with overhead growing with constraint complexity, and reasoning inside JSON keeps constraints active from the start through the end of generation.