Structured outputs are model responses that are constrained to follow a schema you supply, such as a JSON Schema. In AI engineering, they let calling code parse and validate model output directly, instead of repairing free text that only approximates the format you asked for.
For example, suppose an agent extracts action items from meeting notes. You supply a schema that requires an array of items, each with a task, an owner, and a due date. Every successful response contains that array with those fields and types, so your code can group items by owner right away.
The guarantee covers shape, not truth. The schema cannot tell you whether the owner was read correctly from ambiguous notes.
Build better agents with Arize
Trace, evaluate, and learn. Build agents that work with Arize AX and start tracing your runs today.
Prefer open source? Try Arize Phoenix for self-hosted, open source agent observability.
Why structured outputs matter
Most model output ends up in code: a database row, an API call, or the next step of an agent. If the shape is wrong, the next step breaks before anyone can judge whether the content was right.
Structured outputs are useful for:
- Extracting data: Turn documents, emails, or notes into records with known fields and types.
- Calling tools: Produce tool arguments that match the tool’s declared parameters.
- Simplifying application code: Validate once and proceed, without a repair path for near-miss JSON.
- Separating failures: Tell a formatting failure apart from a reading or judgment failure.
In agent systems, the same mechanism underlies many tool calls. The model’s choice of tool and its arguments is itself a structured output.
For more on how agents call tools, see tool calling.
How structured outputs work
When you supply a schema, the serving system restricts generation so the output stays valid against it. Property names match, types match, required fields appear, and enumerations stay within their allowed values.
This holds only for the subset of schema features the provider supports. OpenAI’s introduction of Structured Outputs describes one implementation of this guarantee and how it relates to function calling. Other providers offer comparable schema-constrained modes, each with its own supported subset and limits.
The developer-facing contract is the same across providers. If the call succeeds, the payload parses against the schema.
That contract changes how you write calling code. Without it, you prompt for JSON and then repair near-misses such as trailing commas, renamed fields, or numbers returned as strings. With it, you validate once and move on.
Structured outputs vs. JSON mode vs. asking for JSON
Three approaches are often confused. They give very different guarantees.
- Asking for JSON in the prompt: An instruction such as “reply with JSON only” is usually followed, but nothing guarantees it.
- JSON mode: Provider JSON modes typically guarantee syntactically valid JSON. They do not guarantee that it contains your fields with your types.
- Structured outputs: Generation is constrained to your schema itself.
Only the third lets you treat a successful response as schema-valid without a repair path. Even then, keep a validator in your code. Schemas, providers, and supported features change, and a check at the boundary costs little.
How to design a schema for structured outputs
Here is the schema for the action-item example. It requires an array of items, and each item has a task string, an owner that is a string or null, and a due date string or null.
{
"type": "object",
"properties": {
"action_items": {
"type": "array",
"items": {
"type": "object",
"properties": {
"task": { "type": "string" },
"owner": { "type": ["string", "null"] },
"due": { "type": ["string", "null"], "format": "date" }
},
"required": ["task", "owner", "due"]
}
}
},
"required": ["action_items"]
}
Notice that owner and due are required but nullable. They are not simply left out of required.
This design forces an explicit “not stated” when the notes name no owner or date. A mandatory string field, by contrast, encourages the model to invent a plausible name or date to fill it.
Some providers’ strict schema modes do not support or enforce the format keyword. This means "format": "date" may be ignored, so validate the date string in your own code.
Validate the response in Python
The following function checks that every item has all three fields and that any due value is a real ISO date. It uses only the standard library.
import json
from datetime import date
def validate_items(raw):
data = json.loads(raw)
errors = []
for i, item in enumerate(data["action_items"]):
for field in ("task", "owner", "due"):
if field not in item:
errors.append(f"item {i}: missing {field}")
due = item.get("due")
if due is not None:
try:
date.fromisoformat(due)
except ValueError:
errors.append(f"item {i}: due is not a date: {due!r}")
return errors
response = """{"action_items": [
{"task": "Send revised pricing", "owner": "Priya", "due": "2026-11-03"},
{"task": "Book the venue", "owner": null, "due": null},
{"task": "Draft the launch post", "owner": "Sam", "due": "next Friday"}
]}"""
for error in validate_items(response):
print(error)
Output:
item 2: due is not a date: 'next Friday'
The third item is valid against the type string, so a provider that ignores format would accept it. Your own check catches it.
What the schema cannot ensure
A schema-valid response can still be wrong. In the action-item example, the schema cannot ensure that:
- The owner was read correctly: Ambiguous notes can lead to the wrong person in a well-typed
ownerfield. - The date was a deadline: A date mentioned in passing can land in
due. - Nothing was left out: An item the notes imply can be omitted entirely, and the array is still valid.
These are semantic questions. You settle them by evaluating against labeled examples, not by tightening the schema.
When a tool call fails in production, “the arguments were missing a required field” and “the arguments named the wrong customer” are different failures. Schema constraint removes much of the first kind. Only evaluation catches the second.
How to evaluate structured outputs
Evaluate in two layers, one for each kind of failure.
The conformance layer is mechanical. Measure what fraction of responses validate against the schema and what fraction parse at all. With true schema-constrained generation, this should be at or near 100%. Any dip points to a provider, schema-feature, or integration problem.
The semantic layer measures correctness. On a labeled set, check how often extracted values are right, how often items are missed or invented, and how errors spread across fields.
The semantic layer uses the ordinary tools of LLM evaluation: labeled datasets, per-field accuracy, and error review. Add it to your evaluation metrics from the start. A format fix that quietly hurts extraction quality is a regression the schema cannot show you.
Use traces to connect bad values to their source
For each call, record the schema version, the raw response, and the validation result. This lets you trace a bad downstream value back to the exact call and prompt that produced it.
It also lets you tell a schema problem from a reading problem after the fact. The Phoenix documentation covers recording this kind of per-call detail during local development.
Limitations of structured outputs
- Valid but wrong: Schemas constrain form, not judgment. The typical failure is a well-typed value that misreads the source.
- Large schemas can hurt content: Large or deeply nested schemas can degrade the values themselves, as the model spends effort satisfying structure.
- Provider support varies: Unsupported keywords may be rejected or silently ignored. Test the exact schema against the exact provider.
- Enums force a choice: An enumeration makes the model pick a value even when none fits, unless you include an “other” option or a nullable path.
- Schemas need maintenance: A schema is an interface contract. Changing it changes downstream behavior, so version it and rerun semantic evaluation when it changes.
Keep schemas as small as the task allows. Every required field is something the model must produce.
Frequently asked questions
Do structured outputs guarantee correct answers?
No. They guarantee that a successful response conforms to your schema, with the right fields and the right types. Whether the values are accurate and complete is a separate question that semantic evaluation on labeled examples answers.
What is the difference between structured outputs and JSON mode?
JSON mode generally guarantees only syntactically valid JSON. Structured outputs constrain generation to your specific schema, so required fields, types, and allowed values are enforced. Asking for JSON in the prompt guarantees neither.
Why make a field nullable if it is required?
It forces the model to state explicitly that information is absent. Without it, the model may omit the field or invent a value. A visible “not stated” is something your code and evaluators can count.
Do I still need to validate structured outputs in my code?
Yes. Providers support different schema subsets, and some ignore keywords such as format. A validator at the boundary also protects you when a schema or provider changes.
How should I test a schema change?
First check conformance, meaning responses still validate. Then rerun semantic evaluation on a fixed labeled set to confirm extraction quality did not regress. Review field-level errors before and after the change.
Putting structured outputs to work
Start with the smallest schema that serves the task. Make fields required but nullable where the source may not contain the value, and add an “other” option to enums that might not fit.
Version the schema like an API. Keep a validator at the boundary, and record the schema version with each call.
Then measure both layers on the same labeled set: conformance rate and per-field correctness. Tools such as Arize Phoenix can capture each call as a trace, so you can review the raw response next to the value that reached your application.
