OpenRouter tests three routes to structured LLM output: 209 of 216 replies met a strict schema, and all 7 misses came from one provider that ignores it
Synopsis
OpenRouter compared three ways to get reliable JSON out of models, using 24 messy support messages across nine models that support structured outputs: send a strict JSON Schema, force a single tool call, or switch to a decision model that only returns labels. Across the nine models, 209 of 216 replies matched a strict schema, and all 7 misses came from one provider that ignores the schema; asking for JSON only in the prompt produced 18 failures, each from a code fence. For GLM 5.3 FlashX, which only supports tools, forcing the tool call gave 22 valid outputs out of 24, against 0 when the schema was sent directly.
Interpretation
Getting a model to emit JSON that code can read directly now has three selectable routes, chosen by what a model's endpoints actually support, instead of a separate patch for each model. The common earlier approach asked for JSON in the prompt, and models occasionally wrapped it in a code fence, added an unrequested field, or wrote a number as text, breaking downstream code. On 24 messy support messages, nine models supporting structured outputs returned 209 schema-valid replies out of 216, with all 7 misses from one provider that ignores the schema; asking for JSON only in the prompt failed 18 times, every time because of a code fence.
The method hands the same schema to the model either as a response-format constraint on the message text or as the parameters of a single tool the model is forced to call, with the code reading the arguments and never running the tool. Strict mode requires every key to be listed in required and every object to set additionalProperties to false, with possibly missing values made nullable rather than omitted; OpenRouter reports that a schema following these strictest rules worked on every model tried. GLM 5.3 FlashX, which only supports tools, returned 0 valid outputs out of 24 when the schema was sent directly, and 22 out of 24 when tool_choice was set to required to force the call, with the two failures putting the string null in order_id.
Models that only support tools can still produce structured output, but whether a provider actually enforces the schema has to be checked, because a request can return 200 with non-conforming content. Structured-output support belongs to each endpoint rather than to the model, so one model can have endpoints that support it and endpoints that do not; by default the request still goes through and the schema is silently ignored. Setting require_parameters to true in the provider object makes the request fail with a 404 when no endpoint supports the sent parameters; GLM 5.3 FlashX's endpoint listing marked required as unsupported, yet every required call that day succeeded, so the field is only a guide.
Perspective
The findings come from 24 support messages, nine models that support structured outputs plus one tool-only model, GLM 5.3 FlashX, with a single run per comparison, and apply to engineering settings that read fields or labels out of model replies.
Whether a provider enforces the schema is not stable, and the supports_tool_choice field in endpoint listings can disagree with actual behavior; Response Healing fixes syntax without reading the schema and does not apply to streamed requests or tool-call arguments; decision-model probabilities drift slightly between runs, so arithmetic judgments such as a refund threshold should be tested on your own data first.
