Structured Output and Constrained Decoding: Making LLMs Return Valid JSON

Every component downstream of a language model expects a data structure. The tool router needs a function name and an argument object. The model emits a sequence of tokens — text — and text is only a data structure if something guarantees it is. Most production systems close that gap with a json.loads inside a try, a regex that strips code fences, and a retry counter. That code exists because the guarantee is missing.

Ask politely in the prompt and a capable model gives you valid JSON most of the time. High-90s reliability sounds like a pass; it is not. Two percent failure over ten thousand calls a day is two hundred broken requests, every day, forever. The arithmetic worsens when you chain calls: an agent making eight model calls per trajectory, each 98 percent parseable, completes cleanly about 85 percent of the time. And the failures cluster on inputs that are unusual, long, or adversarial — exactly where you most want the system to behave. "Usually valid" is not a foundation for a control plane.

There is a way to make invalid output not merely unlikely but impossible. Getting there means climbing a ladder, because each rung explains what the next one fixes.

The ladder

Rung 1: prompt and hope. Ask for JSON in the instructions, maybe add an example, parse what comes back. The guarantee is zero: nothing in the decoding process is aware that you asked. The failure shapes are a taxonomy: the JSON wrapped in a fenced markdown block, because that is what JSON looks like in training data; a preamble sentence; commentary after the closing brace; single quotes; trailing commas; unescaped newlines and quotes inside strings; and, on long outputs, truncation at the token limit. Each is patchable; each patch is a heuristic that fails on input you have not seen.

Rung 2: retry with validation feedback. Parse; on failure, feed the error back and ask again; cap the attempts. This works, and it is the right move if you are shipping this week: models are good at fixing a specific error they have been shown. It is also error recovery rather than prevention. The cost is extra calls, latency measured in whole generations, and an unbounded tail — there is no N for which "N retries" is a guarantee, only a probability that keeps shrinking.

Rung 3: provider structured-output modes. Most hosted providers now offer a mode where you supply a schema and the service enforces it during generation. Where available this is a strong guarantee and nearly free to adopt — a request parameter, not a component in your stack. Two constraints: portability, since schema format and enforcement semantics differ between providers, making this a sticky form of provider coupling; and coverage, since these engines typically support a restricted subset of JSON Schema, with recursion, some composition keywords, string patterns, and numeric bounds the usual casualties.

Rung 4: constrained decoding. You enforce the grammar yourself, inside the decoding loop — essentially what the provider modes do internally. Running it yourself means it works with open weights and with formats that are not JSON at all, and the guarantee is structural rather than statistical.

How constrained decoding actually works

At each step the model produces a vector of logits — one score per token in the vocabulary, tens of thousands to a few hundred thousand entries. Sampling turns that vector into the next token, and everything interesting happens in the gap between those two operations.

A constraint engine sits in that gap. It holds a state machine derived from your schema or grammar and tracks where generation sits inside it. Before sampling it asks: given everything emitted so far, which tokens could legally come next? Every token that could not is masked — its logit set to negative infinity, so its probability after softmax is exactly zero.

State the consequence precisely, because it is the whole point. The model is not being persuaded to stay in format, and it is not checked afterwards: tokens that would break the format are removed from the space of possible outputs before the choice is made, and a sampler cannot select a token with zero probability. Invalid output is not improbable. It is unreachable.

Walk a short one. The schema is an object with one required string field named verdict.

Nothing emitted. The only legal first character is an opening brace, so every token that does not begin one is masked. The model emits {.

After the brace, two continuations are legal in general: a double quote opening a key, or a closing brace if the object may be empty. This object has a required field, so the brace is masked too — only quote-initial tokens survive.

Inside the key, the machine knows the permitted key set. If verdict is the only one, every token that does not continue toward verdict plus its closing quote is masked. Note where tokenizer detail bites: one token may carry several characters, possibly including that quote, so the engine reasons over tokens, not characters.

Key closed. The machine expects a colon, then optional whitespace, then a value. The field is typed string, so the only legal opener is a double quote: every digit token, every boolean and null literal, the opening bracket and brace are masked, whatever the model would have preferred.

Inside the string the mask is at its widest, nearly the whole vocabulary, with two exceptions: a raw control character or an unescaped quote would break the JSON, and a backslash moves the machine into an escape state where only valid escape characters survive.

String closed. Every required field is satisfied, and if the schema admits no further properties the closing brace is the only legal continuation. Critically, the end-of-sequence token stays masked until the document is complete, so the model cannot stop halfway.

That is the entire mechanism — a cursor in a state machine, a legality query per step, a mask applied to the logits.

Grammars, schemas, and how one becomes the other

JSON Schema is the common interface because it is what your team already writes and your validation library already understands: it describes a data shape. A context-free grammar is the more general object. In a notation such as GBNF, used by several inference engines, a grammar defines a language through production rules:

root  ::= object
object ::= "{" ws pair ("," ws pair)* ws "}"
pair  ::= string ws ":" ws value
string ::= "\"" char* "\""

Grammars are not restricted to JSON. Anything with a formal syntax can be constrained this way: a SQL dialect narrowed to the statements you are willing to execute, a structured log line, a domain-specific command language. Regular expressions are the same idea at lower expressive power.

Compilation runs in two stages. The schema is lowered into a grammar: each type becomes a production, each enum an alternation of literals, required properties an ordered sequence. The grammar is then compiled against the tokenizer of the model you serve, because legality is a property of tokens — two models with different vocabularies need different masks for the same grammar.

The naive implementation, testing all N vocabulary tokens against the parser at every step, is too slow to ship. Practical engines precompute: they build an automaton over the grammar and index the vocabulary against its states, so the mask for the current state is a lookup and a bitwise operation rather than a scan. That index is built once per schema-and-tokenizer pair and reused by every request sharing it, which is why per-token overhead in a good engine is small next to the forward pass.

What it costs

Compilation takes real time — milliseconds for a flat object, longer for something large or recursive — paid once per unique schema, so cache it and warm it at startup.

Per-step masking is a lookup and a mask application with a prebuilt index; without one it shows up as reduced tokens per second.

Coverage varies: recursion, composition keywords, string patterns and numeric bounds are the usual friction points, and a schema that validates in your test suite may be rejected or silently simplified at compile time.

Against that, constrained decoding is frequently faster end to end than the retry loop it replaces. A retry discards an entire generation — every token you paid for and the latency of producing them — and constrained decoding never produces one you would have discarded. Retry even a few percent of calls and the masking overhead is bought back immediately.

Structurally valid is not semantically correct

This is the part that gets skipped, and the part that produces incidents.

Constrained decoding guarantees the bytes parse and the shape matches. It says nothing about whether the values are right. A confidence of 0.95 is valid JSON. So is a severity of critical on a benign event, a vulnerability identifier with the correct format and no referent, a customer ID that is well-formed and belongs to someone else. The schema was satisfied; the output is wrong. You removed the loud failure mode and kept the quiet one.

Worse, over-constraining actively degrades quality, in two ways.

Forced fields invite fabrication. If a field is required and the model has no basis for a value, the mask will not let it stop and will not let it emit nothing. Something must be produced, so something plausible is produced. You have engineered a system that cannot decline.

A schema that excludes uncertainty guarantees you never receive it. If your enum is approve and deny, you will never get insufficient_information, however insufficient the information is. The model's uncertainty had nowhere to go, so it was resolved into one of your two options, and you cannot tell afterwards which answers were confident.

Design against this. Make optional fields genuinely optional — nullable types, not required fields carrying sentinels. Put an unknown or insufficient_evidence member in every enum where ignorance is a possible true state, and include a refusal path, so declining is a representable output rather than a protocol violation. Then validate semantics separately, after parsing: does this identifier exist, is this date plausible, does the cited evidence appear in the input at all.

Schema design that works with the model

The schema is a prompt: the model reads field names, descriptions, and ordering, and they shape output as much as your instructions do.

Flat beats deeply nested. Every level is more structure to track and another place to lose the thread; two levels is comfortable, five is asking for trouble.

Field names carry meaning. exploitability_assessment produces different content than field_3, and one line of description on an ambiguous field often beats a paragraph of prompt instruction, because it sits next to the point of generation.

Enums beat free text for anything categorical. An enum is enforceable, comparable, and monitorable; a free-text severity field accumulates High, high, and high (but see note).

Keep required fields genuinely required. Each one is a field the model must produce for every input, including the inputs where it should not.

Order fields so reasoning comes before conclusions

A model generates left to right, each token conditioned on the tokens before it. Intermediate text is not decoration; it is the substrate later tokens are computed over. That is the entire reason chain-of-thought prompting works.

So a schema that puts verdict first and reasoning second has committed the model to an answer before it generated a token of analysis. What follows is not the reasoning that produced the verdict; it is a justification conditioned on a conclusion already fixed. You get a worse answer plus a convincing rationalisation of it. Put reasoning-bearing fields first: evidence, analysis, classification, confidence. Property order is yours to set in a constrained schema, so this is a one-line change with a measurable effect on quality.

Reasoning and structure together

A schema is a cage and thinking wants room. Three resolutions, in rough order of cost.

A reasoning field inside the schema, placed first: one call, thinking captured in the object you already store. The limit is that it is still constrained text inside a JSON string, produced while the model tracks format.

Two calls, think then extract: the first runs unconstrained, natural prose; the second takes that output and does nothing but populate the schema. It costs a round trip, and extraction can misread the analysis, but the analysis itself is unimpaired.

Reasoning outside the constrained span: the model thinks in a region the constraint is not applied to, and only the structured portion is generated under the mask — most of the two-call benefit inside one call, where your stack supports it.

The security angle

Constrained output is a genuine control surface, and underused as one. When model output feeds a tool call, a query builder, a shell command, or another agent, that output is an instruction to a downstream system, and a grammar bounds what it can be. A model that has been prompt-injected, fed poisoned retrieved content, or is simply confused still cannot emit a token the grammar disallows. If your SQL grammar permits only single-statement reads against three named tables, a compromised generation cannot emit a destructive one. Blast radius is set by the grammar, not by the model's behaviour in the moment.

Now the limit, plainly, because the failure here is believing more than is true. Constrained decoding is a containment measure, not an injection defence: it constrains the shape of the output, never the intent behind it. An attacker who cannot make the model emit arbitrary text can still make it choose the wrong enum member or pass an attacker-chosen identifier into a legal field. Every value remains untrusted input, and authorisation, bounds checks, and ownership checks happen after parsing, before you act. Keep the grammar as narrow as the task allows, so the legal action space stays small and enumerable.

Practical guidance

Enforce as close to the sampler as you can. Engine-side or provider-side enforcement is a guarantee; a validation library wrapped around the response is a detector. Validate client-side anyway — it catches subset gaps and engine bugs.

Test with the inputs that hurt. Empty input. Input that is itself a refusal. Input carrying an injection aimed at the field you care about. Input long enough to push generation against the token limit. The question is not whether the output parses — it will — but whether the values are sane when the input is not.

Log compilation failures with everything: the schema as submitted, the engine, the unsupported construct, the triggering request. Fail loudly — a silent fallback to unconstrained generation converts a guarantee into a coin flip without changing a line of your logs.

Handle truncation explicitly. The token limit is the one way a constrained generation still ends badly: the engine masks the stop token until the document is complete, but it cannot manufacture budget, so you get a valid prefix of a valid document — an invalid one. Detect this from the finish reason rather than the parse, size the limit against the worst case the schema permits, and cap array lengths and string sizes in the schema.

A decision checklist

• Does anything downstream parse this output? Then structure is a requirement, not a nicety.
• Is a provider structured-output mode available, and does its subset cover your schema? Start there.
• Open weights, or a non-JSON format? Engine-side constrained decoding with a grammar.
• Can you compile the schema once and cache it? Dynamic per-request schemas are the first thing to fix.
• Does the schema permit "I don't know" — nullable field, unknown enum member, refusal path? If not, you get fabrication.
• Do reasoning-bearing fields come before conclusions in the declared order?
• Is there a semantic validation layer after parsing that checks values rather than shapes?
• Does the output feed a tool, query, or command? Then every value is untrusted; authorise before acting.

Get the first four right and malformed output stops being a category of incident. Get the rest right and you avoid trading it for a quieter, worse one.