Structured Outputs Guide

TL;DR

• Free text fails for machines. Prose, markdown fences, and “helpful” commentary break parsers. Applications need typed fields, enums, and bounded lists — not essays.

• Three enforcement layers exist. Prompt-only JSON (no guarantee), JSON mode (valid syntax), and schema-constrained decoding (valid against your JSON Schema). Prefer the strongest layer your provider supports.

• Prompting and schema enforcement are different jobs. Prompt engineering biases what the model means; structured outputs constrain what it is allowed to emit. Use both.

• Pydantic (or equivalent) is the application contract. Define types once, export JSON Schema to the API, validate every response, retry with error context, then act.

• Constrained decoding is not correctness. It can force shape and types while still producing wrong categories, hallucinated entities, or injection strings inside valid fields. Pair with guardrails and evaluation.

• Structured outputs power extraction, classification, UI forms, and function calling. Tool arguments are structured outputs with an extra tool-selection step.

On this page

• Why This Matters
• The Problem Structured Outputs Solve
• How We Got Here
• What Are Structured Outputs?
• How Structured Outputs Work
• Architecture
• Step-by-Step Flow
• Real Production Example
• Design Decisions
• Comparisons
• Common Mistakes
• Where It Breaks Down
• When NOT to Use Structured Outputs
• Running in Production
• Related Guides
• Interview Questions
• Key Takeaways
• FAQs
• References
• Further Reading

Why This Matters

Every production LLM path eventually hits the same wall: the model returns helpful text, but your application needs typed data. Classify a support ticket. Extract entities for a knowledge graph. Fill a form. Emit SQL parameters. Hand an agent a tool invocation.

Without structured outputs, engineers invent parsers — strip markdown fences, regex for braces, retry when JSON is malformed, and silently fail when the model adds a paragraph of explanation. That is not integration; it is hoping the next sample lands in a lucky region of the output distribution.

Structured outputs replace hope with a contract. The difference between a demo (“look, it returned JSON!”) and a system (“every response validates against TicketClassification before routing”) is whether machine consumers can rely on shape, types, and required fields.

This guide sits next to prompt engineering (how you ask), function calling (structured tool invocations), guardrails (policy beyond schema), and evaluation (whether the pipeline still works after a model or schema change).

The Problem Structured Outputs Solve

An LLM samples the next token from a probability distribution. Nothing in that sampling loop knows your Python types, your database columns, or your API contract. Left unconstrained, the model optimizes for fluent, helpful-looking text — which is exactly wrong for machine consumption.

Typical free-text failure modes:

Model output:
"Here's the classification:
```json
{"category": "billing", "priority": "high"

Let me know if you need anything else!"

Your parser breaks on markdown wrapping, trailing commas, missing braces, extra prose, wrong types (`"urgent"` vs an enum), hallucinated fields, and truncated objects when `max_tokens` cuts the stream mid-JSON.

| Failure | Why it happens | What breaks |
|---|---|---|
| **Markdown / prose wrapper** | Chat models are trained to be helpful narrators | `json.loads` fails |
| **Invalid JSON syntax** | Open sampling has no grammar | Parser exception |
| **Wrong shape** | Prompt said “JSON” but not which keys | `KeyError` / null deref |
| **Wrong enum / type** | Model invents a plausible label | Bad routing / DB write |
| **Truncation** | Output budget exhausted mid-object | Incomplete payload |
| **Silent semantic error** | Shape is valid; meaning is wrong | Corrupt downstream state |

Structured output modes address the **syntax and shape** failures at generation time. Application validation and business rules address the **semantic** failures afterward. Neither layer alone is enough.

## How We Got Here

Structured generation matured as teams moved from chat demos to production pipelines that write to databases and call APIs.

**Diagram: Evolution of structured LLM outputs**

```mermaid
timeline
    title From hope-based JSON to constrained decoding
    2022-2023 : Prompt "return JSON"
              : Regex and fence stripping
    2023 : Provider JSON mode
         : Valid syntax, any shape
    2024 : Schema-constrained APIs
         : OpenAI structured outputs
    2024-2026 : Pydantic + Instructor defaults
              : Tool args as structured outputs

Teams moved from prompt-only JSON through syntax guarantees to schema-level constrained decoding — then still kept application validation.

Function calling evolved in parallel: the model selects a tool and emits typed arguments. Response structured outputs and tool arguments share the same underlying idea — constrain the token stream to a schema — with different product surfaces.

What Are Structured Outputs?

Structured outputs are LLM responses constrained to a predefined format — typically a JSON object with named fields, types, enums, and nesting rules. Instead of open-ended prose, the model fills a template your code can validate.

from pydantic import BaseModel, Field
from typing import Literal

class TicketClassification(BaseModel):
    category: Literal["billing", "technical", "account", "other"]
    priority: Literal["low", "medium", "high", "critical"]
    summary: str = Field(max_length=200)
    suggested_assignee: str | None = None
    confidence: float = Field(ge=0.0, le=1.0)

Pass TicketClassification.model_json_schema() (or a provider-wrapped form) to the LLM API. Receive JSON. Validate with TicketClassification.model_validate_json(...). Reject or retry on failure. Only then route, write, or call an API.

Prompting vs schema enforcement

Use prompt engineering to teach what good answers look like. Use structured outputs to make invalid shapes impossible (or vanishingly rare). Treat them as complementary, not substitutes.

Constrained decoding vs prompt-only JSON

Prompt-only JSON asks the model to cooperate. Sampling remains open; the model can still emit prose or broken braces.

Constrained decoding (grammar / JSON Schema / FSM / CFG masking) intersects the model’s next-token distribution with tokens that keep the partial output inside a valid language. At each step, illegal tokens are masked or given zero probability. The result is a string that is valid JSON and, in schema mode, matches your schema’s structure.

Source: OpenAI — Structured Outputs

Important

Constrained decoding guarantees form, not truth. A field typed as string can still contain a SQL injection. An enum can still pick the wrong category. Always validate business rules and sanitize before side effects.

How Structured Outputs Work

Layer stack

1. Schema definition — Pydantic model, Zod schema, or hand-written JSON Schema. Single source of truth.
2. Provider encoding — API wraps the schema (response_format, tool parameters, grammar).
3. Constrained generation — Decoder samples only tokens that keep the output valid (when supported).
4. Transport — Raw JSON string or provider-parsed object in the response.
5. Application validation — Re-parse with your typed model; reject mismatches.
6. Business checks — Confidence thresholds, allowlists, PII rules, guardrails.
7. Side effects — DB writes, API calls, agent tool execution — only after steps 5–6 pass.

JSON mode vs schema mode (mechanically)

from openai import OpenAI

client = OpenAI()

response = client.chat.completions.create(
    model="gpt-4o",
    messages=[
        {"role": "system", "content": "Classify support tickets."},
        {"role": "user", "content": "I was charged twice for my subscription!"},
    ],
    response_format={
        "type": "json_schema",
        "json_schema": {
            "name": "ticket_classification",
            "strict": True,
            "schema": TicketClassification.model_json_schema(),
        },
    },
)

raw = response.choices[0].message.content
ticket = TicketClassification.model_validate_json(raw)

Warning

JSON mode guarantees valid JSON — not valid business data. A priority field can still contain "urgent" when your schema allows only low|medium|high|critical. Always validate after generation.

Model limitations vs engineering mitigations

Libraries such as Instructor, Outlines, and framework wrappers (LangChain / LlamaIndex structured output helpers) adapt provider APIs and implement retry-on-validation-error. Prefer understanding the underlying contract over treating the library as magic.

Source: Pydantic — Models

Architecture

In production, structured outputs are a pipeline with observability and a hard gate before side effects — not a single API flag.

Diagram: Structured output pipeline

flowchart TB
    subgraph define [Define]
        Model[Pydantic / Zod schema]
        Schema[JSON Schema export]
        Model --> Schema
    end
    subgraph generate [Generate]
        Prompt[Prompt + few-shot]
        API[LLM API constrained decode]
        Prompt --> API
        Schema --> API
    end
    subgraph verify [Verify]
        Val[Typed validation]
        Biz[Business rules / guardrails]
        Val --> Biz
    end
    API --> Val
    Biz -->|pass| Act[DB / API / tool]
    Biz -->|fail| Retry[Retry with error context]
    Retry --> API
    Biz -->|exhausted| Queue[Human review / dead letter]

Schema in, constrained generation, typed validation, then side effects — retries feed validation errors back into the next attempt.

Step-by-Step Flow

Diagram: Request lifecycle for schema-constrained generation

sequenceDiagram
    participant App as Application
    participant Reg as Schema registry
    participant LLM as LLM API
    participant Val as Validator
    participant Sink as Downstream

    App->>Reg: Load schema version
    App->>LLM: Prompt + JSON Schema (strict)
    LLM-->>App: Candidate JSON
    App->>Val: model_validate_json
    alt Valid
        Val-->>App: Typed object
        App->>Sink: Side effect
    else Invalid
        Val-->>App: ValidationError
        App->>LLM: Retry with error context
        LLM-->>App: Fixed JSON
        App->>Val: Re-validate
    end

Every successful path validates before the sink; exhausted retries go to review, not silent parse fallbacks.

1. Define the contract — Pydantic / Zod model with enums, bounds, and optional fields explicit.
2. Export schema — model_json_schema() or equivalent; strip unsupported keywords for the target provider.
3. Compose the prompt — Short system instructions; put untrusted input in the user role; add 1–3 few-shot examples for hard edge cases.
4. Call with constraints — json_schema / strict mode, tool parameters, or local grammar (Outlines / llama.cpp).
5. Validate — Typed parse; never trust provider “parsed” objects blindly if your app has richer constraints.
6. Apply business rules — Confidence floors, max entities, allowlists.
7. Retry or escalate — Feed ValidationError text back once or twice; then human queue or simplified schema — never regex free text as fallback.
8. Log — schema_version, model_id, validation outcome, retry count, latency, token usage.

Real Production Example

OpenAI structured outputs + Pydantic

Support-ticket classification with strict JSON Schema and application validation:

from __future__ import annotations

import os
from typing import Literal, TypeVar

from openai import OpenAI
from pydantic import BaseModel, Field, ValidationError

client = OpenAI(api_key=os.environ["OPENAI_API_KEY"])
T = TypeVar("T", bound=BaseModel)


class TicketClassification(BaseModel):
    category: Literal["billing", "technical", "account", "other"]
    priority: Literal["low", "medium", "high", "critical"]
    summary: str = Field(max_length=200)
    suggested_assignee: str | None = None
    confidence: float = Field(ge=0.0, le=1.0)


def extract_structured(
    user_text: str,
    schema: type[T],
    *,
    max_retries: int = 2,
    timeout_s: float = 30.0,
) -> T:
    system = (
        "Classify the support ticket. "
        "Use only the allowed enum values. "
        "Set confidence from 0 to 1 based on clarity of the request."
    )
    messages = [
        {"role": "system", "content": system},
        {"role": "user", "content": user_text},
    ]
    last_error: Exception | None = None

    for attempt in range(max_retries + 1):
        response = client.chat.completions.create(
            model="gpt-4o",
            messages=messages,
            temperature=0,
            timeout=timeout_s,
            response_format={
                "type": "json_schema",
                "json_schema": {
                    "name": schema.__name__,
                    "strict": True,
                    "schema": schema.model_json_schema(),
                },
            },
        )
        raw = response.choices[0].message.content or ""
        try:
            return schema.model_validate_json(raw)
        except ValidationError as e:
            last_error = e
            if attempt == max_retries:
                break
            messages.append(
                {
                    "role": "user",
                    "content": (
                        f"Previous output failed validation:\n{e}\n"
                        "Return corrected JSON only."
                    ),
                }
            )

    raise RuntimeError(f"Structured extraction failed: {last_error}")


ticket = extract_structured(
    "I was charged twice for my subscription!",
    TicketClassification,
)
# Only now: route, enqueue, or write to DB
assert ticket.category in {"billing", "technical", "account", "other"}

Instructor-style pattern

Instructor wraps the same contract — schema export, provider adapters, and retry-on-validation-error — behind a typed call:

# Conceptual Instructor-style usage (API surface varies by version)
import instructor
from openai import OpenAI

client = instructor.from_openai(OpenAI())

ticket = client.chat.completions.create(
    model="gpt-4o",
    response_model=TicketClassification,
    messages=[
        {"role": "user", "content": "I was charged twice for my subscription!"},
    ],
    max_retries=2,
)

Whether you use raw OpenAI APIs or Instructor, keep the same Pydantic model as the source of truth for prompts, API schemas, and validation.

Multi-stage entity extraction

For longer documents, do not ask for a 40-field nested object in one shot:

class Entity(BaseModel):
    name: str = Field(max_length=120)
    type: Literal["Person", "Organization", "Product", "Location"]
    confidence: float = Field(ge=0.0, le=1.0)


class ExtractionResult(BaseModel):
    entities: list[Entity] = Field(max_length=50)
    source_span: str = Field(max_length=500)


def extract_entities(chunk: str) -> ExtractionResult:
    result = extract_structured(
        f"Extract named entities from:\n\n{chunk}",
        ExtractionResult,
    )
    result.entities = [e for e in result.entities if e.confidence >= 0.7]
    return result

Only validated, high-confidence entities reach the graph store. Persistent validation failure goes to a review queue — never silent corruption.

Design Decisions

Common patterns

• Extract-then-compute — Model returns raw fields; application computes totals, IDs, and joins.
• Classify-then-route — Enums drive queues; never open str for categories.
• Tool-arg schemas — Same structured-output discipline for agent tools.
• Schema versioning — Stamp schema_version on stored records; migrate readers before writers.
• Partial structured UI — Free-text answer for users + parallel structured block for machines when both are needed.

When should I use this?

Comparisons

Enforcement strength:

Response schema vs function calling:

Python ecosystem (illustrative):

Common Mistakes

1. Trusting JSON mode without validation — Valid JSON ≠ valid business data. Always run typed validation.
2. Schemas that are too complex — Deep 30-field trees fail more often. Decompose into stages.
3. No retry on validation failure — Feed the error back once or twice; most recoverable failures clear.
4. Open string categories — Prefer Literal / enums. Open str invites invented labels.
5. Ignoring token and schema size — Large schemas consume prompt budget. Keep descriptions tight; bound lists.
6. Parsing free text as fallback — If structured mode fails, do not regex prose. Retry, simplify, or escalate.
7. Asking the model to compute derived fields — Extract inputs; compute total, hashes, and joins in code.
8. Skipping evaluation — Schema compliance ≠ field accuracy. Measure both on a golden set.
9. Treating constrained decoding as a guardrail — Shape safety is not content safety.

Where It Breaks Down

• Inherently unstructured tasks — Creative writing, nuanced analysis, and long-form answers degrade when forced into JSON. Prefer free text or a hybrid (prose + small metadata object).
• Rapidly changing schemas — Without versioning, producers and consumers drift. Migrate deliberately.
• Provider gaps — Some models only offer JSON mode or tool-use-as-structure. Reliability drops; compensate with validation and retries.
• Unsupported schema features — Exotic JSON Schema keywords may be ignored or rejected. Stay within the provider’s supported subset and test.
• Long documents — Single-pass extraction hits truncation and attention limits. Chunk and merge with deterministic code.
• Semantic hallucination inside valid fields — Entities and categories can be confidently wrong while validating perfectly. See Hallucinations.

When NOT to Use Structured Outputs

• The consumer is a human reading prose — Do not wrap every chat reply in JSON. Use structure only for machine paths.
• You only need a one-off script — Prompt-only JSON may be enough if a human inspects the result.
• The “schema” is really a workflow — Multi-step decisions belong in code or an agent graph, not one mega-object.
• You need actions, not answers — Prefer function calling when the model must invoke tools.
• Correctness is catastrophic-sensitive — Structure helps parsing; it does not replace verifiers, dual control, or human approval for irreversible actions.

Warning

Never execute side effects on unvalidated LLM output. Constrained decoding is necessary but not sufficient for production safety.

Running in Production

Best Practice

Use the strongest schema constraint available, validate every response with your typed model, cap retries, version schemas, and gate deploys on golden-set field accuracy — not just “is valid JSON.”

Production checklist

• Pydantic / Zod / JSON Schema as single source of truth
• Schema-constrained decoding enabled where supported (strict when available)
• Typed validation on every response before downstream use
• Retry loop (max 2) with validation error feedback
• Enums / Literals for categories; bounds on strings and lists
• Validation failure metrics and alerting (investigate if rate rises)
• Human review / dead-letter queue for persistent failures
• Schema version stamped on stored records
• No DB writes or tool calls until validation passes
• Golden eval set in CI for schema changes and model swaps

Related Guides

Foundations:

• Prompt Engineering — how instructions bias output before constraints apply
• Tokens · Large Language Models — what constrained decoding operates on

Adjacent LLM Concepts:

• Function Calling — structured tool invocations (sibling pattern)
• Tool Calling — agent tool arguments in practice
• Hallucinations — valid shape still wrong content

Reliability & systems:

• Guardrails — policy and safety beyond JSON Schema
• Evaluation — measure field accuracy and regressions
• AI Agents — agents depend on structured tool I/O every step

Tools: ChatGPT · Claude · Gemini · LangChain · Pydantic AI

Interview Questions

1. Why does free-text LLM output fail for machines?
   Models optimize for fluent helpful text. Machines need typed fields and stable shapes. Prose, fences, and invented keys break parsers and corrupt downstream systems.

2. What is the difference between JSON mode and schema-constrained structured outputs?
   JSON mode guarantees syntactically valid JSON. Schema mode additionally constrains fields, types, required keys, and enums to your JSON Schema.

3. How is prompting different from schema enforcement?
   Prompting biases the distribution via instructions; schema enforcement restricts allowed tokens during decoding. Prompting shapes meaning; constraints shape form.

4. Does constrained decoding guarantee correct answers?
   No. It guarantees form (and often types). Categories can still be wrong and strings can still be unsafe. Validate and apply business rules.

5. How do structured outputs relate to function calling?
   Both use schemas for machine-readable objects. Function calling adds tool selection; response schemas shape the final answer.

6. Why still use Pydantic if the API is in strict schema mode?
   Defense in depth: provider bugs, partial support, richer app constraints, and a typed object for the rest of your codebase.

7. What do you do when validation keeps failing?
   Cap retries with error feedback, flatten the schema, add few-shot edge cases, switch to multi-stage extraction, or escalate — do not regex free text.

8. What production metrics matter?
   Validation failure rate, retry rate, field-level accuracy on a golden set, latency including retries, and schema-version mix in stored data.

Key Takeaways

• Free text fails for machines; structured outputs exist to make LLM responses safely consumable by code.
• Prefer schema-constrained decoding over JSON mode over prompt-only JSON; always validate in the application.
• Prompting and schema enforcement solve different problems — use both.
• Constrained decoding is not semantic correctness or content safety; add guardrails and evaluation.
• Flat schemas, enums, bounded lists, retries with error context, and schema versioning are production essentials.
• Function calling is the tool-oriented twin of response structured outputs.

FAQs

What is the difference between JSON mode and structured outputs?

JSON mode guarantees syntactically valid JSON. Structured outputs (schema mode) guarantee the JSON matches your JSON Schema — fields, types, required keys, and enums.

Do I still need Pydantic if I use schema-constrained decoding?

Yes. Constrained decoding dramatically reduces shape errors but is not a substitute for application-layer validation, richer constraints, or typed objects in your code.

How do structured outputs relate to function calling?

Function calling produces structured tool invocations (name + arguments). Response schema enforcement produces structured final answers. Both use JSON Schema; function calling adds tool selection. See Function Calling.

Which providers support schema-constrained generation?

OpenAI (response_format: json_schema with strict mode), many Azure OpenAI deployments, Gemini response schemas, and local models via Outlines / llama.cpp grammars. Anthropic commonly uses tool use for structured data. Check current provider docs — support changes.

How should I handle optional fields?

Use T | None = None (or JSON Schema null unions). Be explicit. Models fill required fields more reliably than optional ones; omit optionals when absence is the common case.

What if validation keeps failing?

Simplify (fewer fields, less nesting), add few-shot examples, use multi-stage extraction, try a stronger model for that step, or send to human review. Do not fall back to free-text parsing.

How do I version schemas?

Stamp stored records with schema_version or a schema hash. Migrate readers before deploying incompatible writers. Include version in logs and eval reports.

Should user-facing answers use structured outputs?

Use structured outputs when a machine consumes the result. For user-facing prose, prefer free text — or a hybrid: prose for the user plus a small metadata object for analytics.

Is constrained decoding the same as guardrails?

No. Constrained decoding enforces format. Guardrails enforce policy, safety, and content rules that schemas cannot express.

How do I evaluate structured output quality?

Track schema compliance and per-field accuracy against gold labels on a fixed set; run it in CI when prompts, schemas, or models change. See Evaluation.

References

• OpenAI — Structured Outputs
• Pydantic — Models
• Instructor documentation
• Outlines — Structured generation
• JSON Schema

Further Reading

• Anthropic — Tool use
• Google AI — Structured output
• OpenAI — Function calling
• LangChain — Structured output