Structured Output, Compared: JSON Mode, Function Calling, and Constrained Decoding

Ask an LLM for JSON and you'll usually get JSON. Ask it for JSON under load, with a genuinely nested schema, or with a field the model has an opinion about, and the three common mechanisms for structured output — JSON mode, function/tool calling, and grammar-constrained decoding — stop being interchangeable. They enforce validity at different points in the generation process, and that difference shows up exactly when you can least afford it: in production, at the tail of the latency distribution, on the inputs that don't look like your test cases.

How the three actually differ

JSON mode (the response_format: json_object style API) constrains the model to emit syntactically valid JSON but says nothing about the shape of that JSON. You still need a schema check afterward, and a plausible-looking-but-wrong object — a string where you expected an enum, a missing required field — will pass the API and fail your code. It's the loosest guarantee and the cheapest to add.

Function/tool calling adds a schema on top: you declare parameters with types, and the provider's backend nudges the model toward matching them, then hands you back a parsed argument object. This catches more shape errors than plain JSON mode, but "nudges toward" is doing real work in that sentence — providers differ in how strictly they enforce the schema, and enum values or required fields can still be violated, especially with longer, more nested schemas.

Grammar-constrained decoding (available in serving stacks like vLLM, in some providers' structured-output modes, and in libraries like Outlines or Guidance) is the strict version: it builds a token-level grammar from your schema and masks any token that would produce an invalid path. Output is guaranteed schema-valid at generation time, not checked after the fact. The tradeoff is that the constraint only knows about *shape*, not *content* — a constrained model can produce a perfectly valid object with a wrong or hallucinated value, because the grammar has no opinion on semantics.

Picking one is a question about where you want the failure

If you're prototyping, or the downstream consumer already validates and retries gracefully, JSON mode is enough — the cost of an occasional malformed object is a retry, not an outage. If you're calling a tool with side effects (charging a card, writing to a database), function calling's schema plus a validation layer before execution is the right amount of ceremony. If you're generating output at high volume where a validation-and-retry loop would meaningfully hurt latency or cost — batch extraction pipelines, high-QPS classification-as-JSON — constrained decoding is worth the setup cost, because it removes retries from the tail entirely.

None of the three replaces checking the *values*, only the *shape*. A constrained decoder will happily emit {"status": "approved"} for a request that should have been denied, with perfectly valid JSON syntax. Structured output solves parsing. It was never going to solve correctness, and treating it as though it does is where most "the JSON was valid but the answer was wrong" incidents start.