Structured Outputs Guide

TL;DR

• Structured outputs force LLMs to return data matching a schema - JSON objects with typed fields instead of prose your code regex-parses.

• Three layers exist: prompt-based JSON, JSON mode (valid JSON guarantee), and schema-constrained decoding (valid against your JSON Schema).

• Pydantic models define the contract - define Python types, generate JSON Schema, validate model output, and fail fast on malformed data.

• Structured outputs power function calling, extraction pipelines, and agent tool args - any time code must consume LLM output programmatically.

• Constrained decoding improves reliability but is not perfect - always validate before database writes, API calls, or side effects.

Why This Matters

Every production LLM integration eventually hits the same wall: the model returns helpful text, but your application needs typed data. Extract entities from a document. Parse a support ticket into category + priority + assignee. Generate a SQL query object. Populate a form.

Without structured outputs, engineers write fragile parsers - regex on markdown fences, retry loops when JSON is malformed, and silent failures when the model adds explanatory prose. Structured output modes replace hope with contracts.

This is the difference between a demo ("look, it returned JSON!") and a system ("every response validates against TicketClassification before routing"). If your LLM output feeds downstream code, databases, or APIs, structured outputs are not optional.

The Problem Structured Outputs Solve

Free-text LLM generation creates integration failures:

Model output:
"Here's the classification:
```json
{"category": "billing", "priority": "high"

Let me know if you need anything else!"

Your parser breaks on: markdown wrapping, trailing commas, missing braces, extra prose, wrong types (`"high"` vs enum), and hallucinated fields.

Structured output modes address this at the generation layer:

1. **JSON mode** - Output is valid JSON (syntax guaranteed).

2. **Schema mode** - Output conforms to your JSON Schema (structure guaranteed).

3. **Function/tool calling** - Output is a typed tool invocation (see [Function Calling](/learn/function-calling)).

Combined with Pydantic validation at the application layer, you get defense in depth.

## What Are Structured Outputs?

Structured outputs are LLM responses constrained to match a predefined format - typically JSON objects with named fields and types. Instead of generating open-ended text, the model fills a template.

```python
from pydantic import BaseModel, Field
from typing import Literal

class TicketClassification(BaseModel):
    category: Literal["billing", "technical", "account", "other"]
    priority: Literal["low", "medium", "high", "critical"]
    summary: str = Field(max_length=200)
    suggested_assignee: str | None = None
    confidence: float = Field(ge=0.0, le=1.0)

Pass TicketClassification.model_json_schema() to the LLM API. The model returns JSON matching this shape. Your code validates with TicketClassification.model_validate(raw).

Structured output pipelines pass a JSON schema to the LLM, receive raw JSON, validate with Pydantic, and route valid responses to downstream code or retry on validation failure.

JSON Mode vs Schema Enforcement

from openai import OpenAI

client = OpenAI()

response = client.chat.completions.create(
    model="gpt-4o",
    messages=[
        {"role": "system", "content": "Classify support tickets."},
        {"role": "user", "content": "I was charged twice for my subscription!"},
    ],
    response_format={
        "type": "json_schema",
        "json_schema": {
            "name": "ticket_classification",
            "strict": True,
            "schema": TicketClassification.model_json_schema(),
        },
    },
)

raw = response.choices[0].message.content
ticket = TicketClassification.model_validate_json(raw)
print(ticket.category)   # "billing"
print(ticket.priority)   # "high"

Warning

JSON mode guarantees valid JSON - not valid data. A priority field can still contain "urgent" when your schema allows only low|medium|high|critical. Always validate with Pydantic or jsonschema after generation.

Pydantic with LLMs

Pydantic is the standard Python layer for structured LLM outputs:

Pydantic models define the expected shape, export JSON Schema to the LLM API, and validate the returned JSON into typed Python objects - retrying with error context when validation fails.

Source: Pydantic

Production Pattern

from pydantic import BaseModel, ValidationError
from typing import TypeVar

T = TypeVar("T", bound=BaseModel)

def extract_structured(
    prompt: str,
    schema: type[T],
    max_retries: int = 2,
) -> T:
    for attempt in range(max_retries + 1):
        response = llm.generate(
            prompt=prompt,
            response_format=schema_to_openai_format(schema),
        )
        try:
            return schema.model_validate_json(response.content)
        except ValidationError as e:
            if attempt == max_retries:
                raise
            prompt += f"\n\nPrevious output failed validation:\n{e}\nFix and retry."

    raise RuntimeError("Unreachable")

Libraries like Instructor, Outlines, and LangChain structured output wrap this pattern with provider-specific adapters. Keep schemas flat when possible - split complex extraction into stages for reliability.

Architecture

Real Production Example

Document entity extraction for a knowledge graph:

class Entity(BaseModel):
    name: str
    type: Literal["Person", "Organization", "Product", "Location"]
    confidence: float = Field(ge=0.0, le=1.0)

class ExtractionResult(BaseModel):
    entities: list[Entity] = Field(max_length=50)
    source_span: str = Field(max_length=500)

def extract_entities(document_chunk: str) -> ExtractionResult:
    result = extract_structured(
        prompt=f"Extract named entities from:\n\n{document_chunk}",
        schema=ExtractionResult,
    )
    # Filter low-confidence before graph merge
    result.entities = [e for e in result.entities if e.confidence >= 0.7]
    return result

Only validated, high-confidence entities reach the database. Failed validation triggers retry or human review queue - never silent corruption.

Decision Matrix

⚠ Common Mistakes

1. Trusting JSON mode without validation - Valid JSON ≠ valid business data. Always run Pydantic validation.

2. Schemas too complex - 30-field nested schemas fail more often. Decompose into stages.

3. No retry on validation failure - Feed the validation error back to the model; one retry fixes most issues.

4. Missing enums - Open str fields invite hallucinated values. Use Literal or enums for categorical fields.

5. Ignoring token limits - Large schema definitions consume prompt tokens. Keep schemas minimal; put descriptions in field docstrings.

6. Parsing free text as fallback - If structured mode fails, do not regex-parse prose. Retry, simplify schema, or escalate to human review.

Where It Breaks Down

Structured outputs struggle when:

• Output is inherently unstructured - Creative writing, nuanced analysis, or long-form content should not be forced into JSON.

• Schema must change frequently - Rapid schema iteration requires versioning and backward-compatible migrations.

• Provider lacks schema mode - Fall back to JSON mode + Pydantic validation; accept lower reliability or use Outlines for local models.

• Fields require computation - Do not ask the LLM to compute total = sum(line_items); extract raw values and compute in code.

For open-ended generation with occasional structure, use prompt engineering to request JSON in specific steps rather than constraining every response.

Running in Production

Best Practice

✅ Best Practices - Instrument every stage, version embedding models, enforce access control at retrieval time, and evaluate on a fixed golden set before shipping changes.

Production Checklist

• Pydantic models (or JSON Schema) as single source of truth for output shape
• Schema-constrained decoding enabled where provider supports it
• Pydantic validation on every LLM response before downstream use
• Retry loop (max 2) with validation error feedback to model
• Enums / Literal types for categorical fields - no open strings
• Field constraints: max_length, ge/le, max_items to bound output size
• Validation failure metrics and alerting (target < 2% failure rate)
• Human review queue for persistent validation failures
• Schema versioning - track which schema version produced each record
• No side effects (DB writes, API calls) until validation passes

Important

Structured outputs are the input layer for function calling and AI agents. Invest in schema design early - it pays off across every tool and extraction pipeline.

Ecosystem

• Pydantic - Python validation and JSON Schema generation

• Instructor - Structured outputs across OpenAI, Anthropic, Gemini with Pydantic

• Outlines - Constrained generation for local/open models

• jsonschema - Language-agnostic schema validation

• Guardrails AI - Validation frameworks with automatic re-asks

• Provider APIs - OpenAI structured outputs, Anthropic tool use, Gemini response schema

Related Technologies

• Function Calling - Structured tool invocations; sibling pattern to response schema enforcement.

• Prompt Engineering - Prompt design for extraction and classification tasks.

• Tool Calling - Agent tool arguments are structured outputs in action.

• Guardrails - Policy validation beyond JSON Schema.

• AI Agents - Agents depend on structured tool calls at every step.

Learning Path

Prerequisites: Prompt Engineering · Function Calling

Next topics: Function Calling · Guardrails · AI Agents

Estimated time: 45 min · Difficulty: Intermediate

FAQs

What is the difference between JSON mode and structured outputs?

JSON mode guarantees syntactically valid JSON. Structured outputs (schema mode) guarantee the JSON matches your JSON Schema - correct fields, types, and required keys.

Do I still need Pydantic if I use schema-constrained decoding?

Yes. Constrained decoding dramatically reduces errors but is not 100% reliable across all providers and models. Pydantic validation is your application-layer safety net.

How do structured outputs relate to function calling?

Function calling produces structured tool invocations (name + arguments). Response schema enforcement produces structured final answers. Both use JSON Schema; function calling adds tool selection.

Which providers support schema-constrained generation?

OpenAI (response_format: json_schema), certain Azure OpenAI deployments, and local models via Outlines/llama.cpp grammars. Anthropic uses tool use for structured data. Check current provider docs - support evolves rapidly.

How do I handle optional fields?

Use Pydantic Optional[T] = None or JSON Schema "type": ["string", "null"]. Be explicit - models fill required fields more reliably than optional ones.

What if validation keeps failing?

Simplify the schema (fewer fields, flatten nesting), add few-shot examples, switch to multi-stage extraction, or use a more capable model for that step.

How do I version schemas?

Include schema_version: Literal["v1", "v2"] as a required field, or tag records at write time with the schema hash. Migrate consumers before deploying schema changes.

Should extraction and generation use the same schema approach?

Use structured outputs for extraction and classification (machine consumption). Use free text for user-facing answers unless the UI requires typed components.

References

• Hugging Face Transformers Documentation
• LangChain Documentation
• OpenAI API Documentation

Further Reading

• Anthropic Documentation
• Google AI for Developers
• LlamaIndex Documentation
• Pinecone Learning Center

Summary

• Structured outputs constrain LLMs to typed JSON - eliminating fragile text parsing. - Use schema-constrained decoding where available; always validate with Pydantic afterward. - Flat schemas, enums for categories, and retry-on-validation-error are production essentials. - Structured outputs underpin function calling, entity extraction, and agent tool arguments.

• Track validation failure rates; persistent failures mean schema or prompt needs simplification. - Never execute side effects on unvalidated LLM output.