Structured outputs are the most reassuring feature in an LLM API. Pass a JSON Schema, get back JSON that fits it, every time. The reassurance is real. It’s also narrower than most people using it think, and the gap is easiest to see if you build the thing.
So I built it: a schema compiler, a token mask and a decoder, in about 200 lines of standard-library Python, then ran it on a real model doing a real extraction task.
How it works
A JSON Schema without recursion describes a regular language. That’s the whole trick. Regular languages are recognised by finite automata, and finite automata can be precomputed and consulted at every decoding step.
Three pieces:
1. Compile the schema to an automaton. Each schema fragment becomes a small graph of states and character transitions. An enum is the union of its literal spellings. An integer is an optional minus, then either 0 or a nonzero digit followed by any digits. An object is {, then its keys and values in order, then }. Fragments nest.
2. Index the vocabulary against the automaton. The question at each step is “which tokens can come next without breaking the schema?” Checking every token one at a time is slow, so the vocabulary goes into a trie, and one walk from the current state covers every token that shares a prefix. The answer is memoised per automaton state. This is the idea in Willard and Louf’s 2023 paper, which became the Outlines library.
3. Mask the logits. At each step, take the model’s scores, zero out everything the index says is illegal, and pick from what’s left. The model never sees the grammar. It just finds that some tokens have become impossible.
options = mask.allowed(state) # token id -> next state
index = torch.tensor(sorted(options))
token = int(index[int(logits[index].argmax())]) # best legal token
state = options[token]In my implementation, indexing a 48,838-token vocabulary took 172 ms once, and the per-token masking cost was 0.03 ms, which is nothing next to a forward pass. That matches the shape of what providers document: Anthropic’s structured outputs page describes compiling the schema into a grammar, extra latency the first time a schema is used, and compiled grammars cached for 24 hours. OpenAI’s guide describes the same first-request delay.
The experiment
The task: extract a service name (one of six), a severity (one of four) and a duration in minutes from one sentence of incident text. Two hundred cases, SmolLM2-135M-Instruct, greedy decoding, the same prompt in every mode.
| Mode | Valid | All fields right | service | severity | minutes |
|---|---|---|---|---|---|
| Prompt only | 3% | 3% | 3% | 20.5% | 86% |
| Constrained to the schema | 100% | 40% | 40% | 100% | 99% |
| Constrained, cut off at 8 tokens | 0% | 0% | 0% | 0% | 0% |
The schema guarantee did exactly what it says. Validity went from 3% to 100%. Two fields were fixed almost entirely:
- Severity went from 20.5% to 100%. Unconstrained, the model wrote
"Medium"and"High"with capitals, which the enum doesn’t allow. The mask made the capital letter impossible, so the model wrote the lowercase one it also knew. Worth checking against your provider, though: Anthropic’s page says the capitalization of enum values isn’t guaranteed and recommends comparing case-insensitively. - Minutes went from 86% to 99%, because the model could no longer wrap the number in a sentence.
And then there’s service, which went from 3% to 40%, and 40% is the ceiling on the whole task. The model doesn’t reliably know which of six service names the sentence is about. When the text says “the ledger team fought a high outage,” the constrained output says "search", cleanly, with perfect syntax.
That’s the thing to internalise. The schema is a constraint on form. The model’s understanding is unchanged. When it doesn’t know, the grammar still forces a legal choice, and the choice comes from the model’s prior rather than the evidence.
The signal you lose
In the unconstrained run, a wrong answer usually looked wrong. The model copied the whole sentence into the service field, or capitalised an enum. A JSON validator would have rejected most of those, and a rejection is a signal: retry, escalate, log.
Under constraint, the same confusion produces a valid, confident, wrong record. Validation errors disappear from your dashboards at the same moment the errors themselves stop being visible. If you turn on structured outputs and your error rate drops to zero, you’ve measured the shape, not the answer.
Two practical consequences:
- Give the model a legal way to say “I don’t know.” Add
"unknown"to enums, or make fields nullable, so uncertainty has somewhere to go other than the most probable wrong value. - Keep an eval that checks field values against ground truth. Schema validity is no longer informative once you’ve guaranteed it.
Two ways it still breaks
Truncation. Cut generation off at 8 tokens and validity was 0%, because the grammar can’t finish a document the decoder isn’t allowed to continue. Both providers document this: Anthropic says output that stops on max_tokens “might be incomplete and not match your schema,” and OpenAI’s guide shows checking for an incomplete status with reason max_output_tokens. Check the stop reason before you parse.
Refusals. A safety refusal is not required to fit the schema. Anthropic returns stop_reason: "refusal"; OpenAI returns a refusal field. Same rule: check the stop reason.
Why maxLength isn’t supported
Anthropic’s page lists what the compiler rejects: recursive schemas, numeric minimum and maximum, string minLength and maxLength, array limits beyond minItems of 0 or 1. It also names a “Schema is too complex for compilation” error. Having built one, I can tell you why.
Bounded repetition can’t be a loop. “At most 40 items” means 40 copies of the item’s states, one per position. I compiled an array of a small object three ways:
| Limit | Unbounded array | Array with maxItems | Nested arrays, both bounded |
|---|---|---|---|
| 5 | 58 states | 142 | 712 |
| 10 | 68 | 332 | 3,322 |
| 20 | 88 | 862 | 17,242 |
| 40 | 128 | 2,522 | 100,882 |
The unbounded array grows by one state per unit of string length and stops. The bounded one multiplies. Nest two and it squares. My first attempt at that table used a limit of 100 and never finished. Providers are drawing the line where their compile budget runs out, and recursion is unbounded nesting, which is why it’s off the list entirely.
One more thing that happened
My first version of the prompt showed the expected shape as {"service":"...","severity":"...","minutes":0}. With that prompt, the model’s answer for minutes was 0 in all 200 cases, constrained or not. The example value had become the answer. Changing the placeholder to <integer> took minutes from 0% to 99%.
I’m including this because it’s the kind of failure structured outputs invite. The output was valid in every one of those 200 cases. Nothing in the pipeline would have complained.
Limits
This is a 135M-parameter model; a frontier model would get the service field right far more often, and the 40% is specific to this setup. My grammar covers a small schema subset, works at the character level and excludes tokens that decode to partial UTF-8, so it can’t produce non-ASCII strings. Production engines compile faster and cover more. The mechanism, and what it does and doesn’t guarantee, is the same.
The code, with 200 randomised masked walks in the test suite that each have to parse and fit the schema: ajinb/constrained-decoding-from-scratch. For what the schema is for in the first place, see function calling.
Comments