Constrained Decoding Sounds Bulletproof. It Only Fixes the Shape.

Your agent asks a tool for "passengers": 2 and gets back "passengers": "two" as a string, so your code throws before the workflow finishes. Or worse: the JSON parses cleanly, every field is the right type, and the customer ID inside it does not exist. Structured outputs and tool calling reliability comes down to two separate problems, and only one of them has an engineering fix.

What's the real difference between structured outputs and tool calling?#

Structured outputs constrain a model's final response so it matches a JSON Schema you supply. Tool calling is the model choosing which function to call and filling its arguments against a schema. Both are schema-shaped generation, and both share the same reliability problem underneath: getting valid, typed data out of a next-token predictor.

A chatbot that extracts {name, email, plan} from a support ticket is a structured-outputs problem: one schema, one response. An agent deciding whether to call search_orders or refund_order, then filling that function's arguments, is a tool-calling problem: pick the right function, then fill it correctly. Most production agents need both in the same loop, choosing a tool and then shaping a clean final response from what it returns.

The distinction matters because the two live in different parts of an API call. Structured outputs shape the model's own text. Tool calling shapes the arguments attached to a function it decided to invoke. If you are still deciding whether to wire a capability in directly or expose it as a shared tool, our breakdown of function calling, tools, and MCP covers that layering question separately from the reliability question this post is about.

Why doesn't an LLM's JSON output always match the schema you asked for?#

An LLM generates text one token at a time from a probability distribution, and a plain-language request for JSON is a suggestion the decoder never enforces. Nothing stops the model from adding a stray field, returning a string where you need an integer, inventing an enum value, or trailing off into a sentence after the closing brace.

The failure list is long and familiar to anyone who has parsed live model output: a required field silently dropped, a count returned as text where you needed an integer, an enum value that is close but not in your allowed set, a whole response wrapped in a markdown code fence, or JSON truncated mid-object because the response hit a token limit. Every one of these is a shape failure, not a content failure. The data model broke before you even got to check whether the answer was any good.

For years the only fix was prompting harder: describe the schema in English, add a one-shot example, tell the model to respond with JSON and nothing else, then wrap the whole call in a parse-and-retry loop for when it still gets it wrong. That approach works often enough to ship, and it is still the fallback for any model or endpoint with no constrained decoding available. It is also fundamentally probabilistic. Every retry costs a round trip, and nothing guarantees the second attempt is any more valid than the first. Stack 3 or 4 retries onto a single step inside an agent loop and a call that should take 2 seconds starts taking 10, with no shape guarantee waiting at the end of it either way.

What is constrained decoding, and how does it force valid shape?#

Constrained decoding compiles your JSON Schema into a grammar, usually a finite-state machine, and masks the model's next-token probabilities at every generation step so only tokens that keep the output on a valid path are ever sampled. The model is not being asked to follow the schema. It is structurally unable to emit a token that would break it.

This is the mechanism behind three products that use different marketing names for the identical idea, and as of August 2026 all three are shipping in production. OpenAI calls its version Structured Outputs: set response_format to json_schema with strict: true, and the schema is compiled into token masks before generation starts. Anthropic calls its version strict tool use and structured outputs: set strict: true on a tool definition and Claude's tool arguments are constrained by grammar-constrained sampling, generally available for Claude 4.5 and later models. Outlines and other open-source libraries build a finite-state-machine index over a model's own vocabulary and apply the same trick to any model you can run weights for, independent of any API vendor.

OpenAI calls it Structured Outputs, Anthropic calls it strict tool use. Same mechanism, different names.

What does constrained decoding actually guarantee, and what does it leave to chance?#

It guarantees the shape: valid JSON, correct types, every required field present, enum values limited to the allowed set. It does not guarantee the shape is filled with anything true. A perfectly schema-valid object can still carry a hallucinated customer ID or an invented date, because a grammar constrains syntax, not facts.

OpenAI's own launch evaluation put a number on that gap: its constrained-decoding model scored 100% on a complex-schema benchmark, against under 40% for the same-generation model without Structured Outputs turned on. The company's own documentation is equally direct about the boundary: Structured Outputs can still contain mistakes, and the fix for those is better instructions or examples, not the schema constraint itself.

Picture a refund tool with a strict schema requiring an order_id string and a reason enum. Constrained decoding guarantees the model returns a real string in that field and a reason drawn from your allowed list. It cannot guarantee order_id is a value that actually exists in your database, or that the reason it picked matches what the customer actually said. The call is completely schema-valid and still fully capable of refunding the wrong order.

That is the honest limit of the whole technique. Masking invalid tokens only removes options that would break the schema; it cannot verify that the token the model picked instead corresponds to something real. The gap is the same one our piece on LLM-as-judge pitfalls describes for eval scores: a well-formed answer is not automatically a correct one, whether the thing grading it is a rubric or a schema.

Every number here is read from a provider's own documentation, not a third-party estimate.

There is a second, subtler cost worth knowing about. Forcing the decoder away from the token it would naturally pick, to satisfy the grammar, can occasionally degrade quality in ways a free-form response would not have shown, because blocking a high-probability token and renormalizing the rest changes what the model effectively treats as its best remaining option. It is a real tradeoff, not a reason to skip constrained decoding, just a reason not to treat it as a quality guarantee it never claimed to be.

Both columns share the same blind spot: neither one guarantees the values are true.

How do OpenAI, Anthropic, and open-source tools compare on what they guarantee?#

OpenAI's Structured Outputs compiles your schema into a token-masked grammar and is available across its current model lineup. Anthropic's strict tool use applies the same grammar-constrained sampling to tool arguments, but excludes a specific list of JSON Schema features. Outlines and similar open-source libraries apply the identical finite-state-machine technique to any model you host yourself, independent of provider.

Anthropic's strict tool use excludes 7 JSON Schema features from its constrained subset, including numeric bounds like minimum and maximum, string-length limits, and recursive schemas. A price field can still come back negative even with strict mode turned on, so you validate that range in your own code regardless of provider. If you are wiring this inside n8n specifically, our walkthrough of n8n's structured output setup covers the node-level configuration this post does not.

Two checks sort every failure: does it pass schema validation, and are the values themselves real.

Is your reliability failure a shape problem or a values problem?#

A shape failure looks like malformed JSON, a wrong type, or a missing field, and that is exactly what turning on strict mode or constrained decoding removes. A values failure is a schema-valid response that is still wrong: a fabricated ID, an impossible date, a plausible but incorrect tool argument. No grammar constraint can check a fact against reality.

Sorting a failure into the right bucket changes what you do next. A shape failure means you turn on your provider's schema mode, or fix a schema that asks for something the constrained subset does not support. A values failure means the schema mode already did its job, and the fix lives one layer up: a lookup against your actual database, a business-rule check, a bounds check on a number, or a human in the loop before anything ships or spends money. Confusing the two wastes effort: tightening a schema that already validates will not touch a hallucinated value, and a new business-rule check will not fix a field that never parses. The 5 signs below cover the values side specifically, since that is the half constrained decoding cannot see.

Every item here parses cleanly. The problem is what is sitting inside the valid shape.

Good tool descriptions help here too, though they work on a different axis: they steer which tool gets picked and what a plausible argument looks like, which heads off some values failures before they happen, but they carry no structural guarantee the way a grammar constraint does. Our guide to writing tool descriptions models actually follow is the companion piece for that half of the problem.

What should you actually build this week?#

Turn on your provider's strict or schema mode first, since it removes an entire failure class for free. Then add your own validation for values, keep a bounded retry loop only for models without constrained decoding, and log shape failures separately from value failures so you know which layer is actually breaking.

That ordering is the whole post. Constrained decoding is not a bigger hammer for prompt engineering; it is a different kind of guarantee that sits underneath it, structural instead of probabilistic, for shape only. Ship it as your floor, keep validating values as your own responsibility, and the two failure modes stop getting confused in your logs, your retries, and your incident reports.