Function Calling Guide

TL;DR

• Function calling lets LLMs output structured JSON specifying which function to invoke and with what arguments - instead of generating free text that your code must parse.

• You define tools as JSON schemas describing function names, parameters, types, and descriptions. The model selects tools and fills parameters based on user intent.

• The execution loop is: send prompt → model returns tool call → your code executes the function → send result back → model generates final response.

• Function calling is the foundation of AI agents - it is how models interact with databases, APIs, code interpreters, and external systems.

• Reliability requires validation - always validate model-generated arguments against your schema before executing.

The model will occasionally produce malformed or incorrect parameters.

Why This Matters

Without function calling, integrating an LLM with your application means parsing free-text output - fragile regex, unreliable JSON extraction, and constant breakage when the model changes phrasing. Function calling replaces this with a structured contract: the model outputs a typed function name and validated arguments; your code executes it.

This is the mechanism behind every AI agent that queries databases, sends emails, executes code, searches the web, or calls APIs. ChatGPT's plugins, Claude's tool use, GitHub Copilot's terminal commands - all function calling under different names.

If you are building anything where the LLM needs to take action (not just generate text), function calling is your integration layer.

The Problem Function Calling Solves

Before function calling (mid-2023), connecting LLMs to external systems required prompt engineering hacks:

Prompt: "Search for the weather in NYC and return the result as JSON."
Model output: "I'll search for the weather. {"city": "NYC", "temp": 72}"
Your code: 😰 parse this somehow

Problems with text-based tool invocation:

• Unreliable parsing - Models wrap JSON in markdown, add explanatory text, or produce malformed output.

• No type safety - Nothing enforces that city is a string or limit is an integer.

• No tool selection guarantee - The model might describe what it would do instead of requesting execution.

• Ambiguous intent - Hard to distinguish "the model is answering" from "the model wants to call a tool."

Function calling solves this by giving the model a dedicated output channel for tool invocations - separate from its text response.

What Is Function Calling?

Function calling (also called tool use by Anthropic, or function invocation) is an LLM capability where the model can output structured data specifying a function to call and its arguments. The model does not execute the function - your application does.

The flow:

ReAct interleaves reasoning traces and tool actions: the model thinks about what to do, calls a tool, observes the result, and repeats until it can answer.

You provide tool definitions as JSON Schema. The model decides whether to call a tool, which tool, and with what arguments. Your code executes the function and returns the result. The model then incorporates the result into its response.

How Function Calling Works

Tool Definition

Tools are defined as JSON Schema objects describing available functions:

tools = [
    {
        "type": "function",
        "function": {
            "name": "get_weather",
            "description": "Get current weather for a city. Use when the user asks about weather conditions.",
            "parameters": {
                "type": "object",
                "properties": {
                    "city": {
                        "type": "string",
                        "description": "City name, e.g. 'New York' or 'London'",
                    },
                    "units": {
                        "type": "string",
                        "enum": ["celsius", "fahrenheit"],
                        "description": "Temperature units. Default: fahrenheit",
                    },
                },
                "required": ["city"],
            },
        },
    },
    {
        "type": "function",
        "function": {
            "name": "search_products",
            "description": "Search the product catalog by name or category.",
            "parameters": {
                "type": "object",
                "properties": {
                    "query": {"type": "string", "description": "Search query"},
                    "category": {"type": "string", "description": "Product category filter"},
                    "limit": {"type": "integer", "description": "Max results", "default": 10},
                },
                "required": ["query"],
            },
        },
    },
]

The description fields are critical - the model uses them to decide when and how to call each tool. Write descriptions as if explaining the function to a colleague.

The Execution Loop

Agent systems extend LLMs with tools, memory, and planning loops so they can take actions in external environments rather than only emit text.

The loop continues until the model produces a text response (no more tool calls) or you hit a maximum iteration limit.

Parallel Function Calls

Modern models can request multiple function calls in a single response:

# Model may return both simultaneously:
tool_calls = [
    {"name": "get_weather", "arguments": {"city": "NYC"}},
    {"name": "get_weather", "arguments": {"city": "London"}},
]
# Execute both in parallel, return both results

Architecture

Multi-agent systems assign specialized roles to multiple LLM agents that communicate, delegate, and coordinate to solve complex tasks.

Source: Research paper

Step-by-Step Flow

1. Register tools - Define function schemas with names, descriptions, and parameter types.

2. Send user message - Include conversation history and tool definitions in the API call.

3. Model decides - LLM either responds with text or returns one or more tool calls.

4. Validate arguments - Check types, required fields, enum values, and business rules.

5. Check permissions - Verify the user is authorized to invoke this tool with these arguments.

6. Execute function - Call the actual implementation (API, database, code).

7. Handle errors - Catch exceptions and format error messages for the model.

8. Return results - Send tool results back to the LLM as tool messages.

9. Model synthesizes - LLM incorporates results into a natural language response.

10. Repeat or return - If the model requests more tools, go to step 4. Otherwise, return the response.

Real Production Example

A customer support agent that queries order data and sends refund requests:

import json
from openai import OpenAI
from dataclasses import dataclass

client = OpenAI()

TOOLS = [
    {
        "type": "function",
        "function": {
            "name": "lookup_order",
            "description": "Look up order details by order ID or customer email.",
            "parameters": {
                "type": "object",
                "properties": {
                    "order_id": {"type": "string", "description": "Order ID, e.g.

ORD-12345"},
                    "customer_email": {"type": "string", "description": "Customer email address"},
                },
            },
        },
    },
    {
        "type": "function",
        "function": {
            "name": "create_refund",
            "description": "Create a refund for an order.

Requires order_id and reason.",
            "parameters": {
                "type": "object",
                "properties": {
                    "order_id": {"type": "string"},
                    "amount": {"type": "number", "description": "Refund amount in USD"},
                    "reason": {"type": "string", "enum": ["duplicate_charge", "defective", "not_received", "other"]},
                },
                "required": ["order_id", "amount", "reason"],
            },
        },
    },
]

TOOL_IMPLEMENTATIONS = {
    "lookup_order": lambda **kwargs: _lookup_order(**kwargs),
    "create_refund": lambda **kwargs: _create_refund(**kwargs),
}

MAX_ITERATIONS = 5

def run_agent(user_message: str, conversation: list[dict] | None = None) -> str:
    messages = conversation or []
    messages.append({"role": "user", "content": user_message})

    for iteration in range(MAX_ITERATIONS):
        response = client.chat.completions.create(
            model="gpt-4o",
            messages=[{"role": "system", "content": "You are a support agent. Use tools to look up orders and process refunds."}] + messages,
            tools=TOOLS,
            tool_choice="auto",
            temperature=0.1,
        )

        message = response.choices[0].message
        messages.append(message)

        if not message.tool_calls:
            return message.content

        for tool_call in message.tool_calls:
            fn_name = tool_call.function.name
            try:
                fn_args = json.loads(tool_call.function.arguments)
            except json.JSONDecodeError:
                messages.append({
                    "role": "tool",
                    "tool_call_id": tool_call.id,
                    "content": json.dumps({"error": "Invalid JSON arguments"}),
                })
                continue

            if fn_name not in TOOL_IMPLEMENTATIONS:
                result = {"error": f"Unknown function: {fn_name}"}
            else:
                try:
                    result = TOOL_IMPLEMENTATIONS[fn_name](**fn_args)
                except Exception as e:
                    result = {"error": str(e)}

            messages.append({
                "role": "tool",
                "tool_call_id": tool_call.id,
                "content": json.dumps(result),
            })

    return "I was unable to complete your request. Please try again or contact support."

def _lookup_order(order_id: str = None, customer_email: str = None) -> dict:
    # Production: query database
    return {"order_id": order_id, "status": "shipped", "total": 49.99, "items": ["Widget Pro"]}

def _create_refund(order_id: str, amount: float, reason: str) -> dict:
    # Production: call payment API
    return {"refund_id": "REF-789", "status": "pending", "amount": amount}

Design Decisions

⚠ Common Mistakes

1. Not validating arguments before execution - The model will occasionally produce wrong types, missing required fields, or out-of-range values. Always validate against your schema and business rules before calling any function.

2. Vague tool descriptions - "Search the database" gives the model no guidance on when to use this tool vs others. Write descriptions that specify trigger conditions: "Search the product catalog when the user asks about product availability, pricing, or features."

3. Too many tools - Providing 50 tool definitions overwhelms the model's selection accuracy. Group related functions or use a two-stage approach: first select a category, then select a tool.

4. No iteration limit - Without a max loop count, a confused model can call tools indefinitely. Always set a maximum (5–10 iterations) and handle the timeout gracefully.

5. Executing without permission checks - The model decides to call delete_user(email="admin@company.com") - your code must verify the requesting user has permission before executing destructive operations.

6. Returning raw API errors to the model - Stack traces and internal error messages leak system information. Return sanitized error messages: {"error": "Order not found"} not {"error": "SQLSTATE[42P01]: relation 'orders' does not exist"}.

Important

Function calling gives the model the ability to invoke your code. Treat every tool call as an API request from an untrusted client - validate, authorize, and sanitize before execution.

Where It Breaks Down

• Tool selection accuracy - With many similar tools, the model picks the wrong one. Mitigate with clear descriptions, fewer tools, and explicit disambiguation in the system prompt.

• Argument hallucination - The model invents parameter values instead of asking the user. Instruct it to request missing information rather than guessing.

• Multi-step dependency failures - If step 2 depends on step 1's result and step 1 fails, the model may proceed with fabricated data. Implement explicit dependency checks.

• Non-deterministic tool selection - The same query may trigger different tools across calls. Acceptable for exploration; problematic for deterministic workflows. Use tool_choice to force specific tools when needed.

• Context consumption - Tool definitions, call arguments, and results all consume context window tokens. A 5-step tool loop can consume 10K+ tokens.

Running in Production

Best Practice

✅ Best Practices - Instrument every stage, version embedding models, enforce access control at retrieval time, and evaluate on a fixed golden set before shipping changes.

Ecosystem

• OpenAI - Function calling via tools parameter. Parallel calls, structured outputs integration.

• Anthropic - Tool use via tools parameter on Claude. Strong instruction following for tool selection.

• Google - Function declarations in Gemini API. Native Google Cloud tool integration.

• LangChain - @tool decorator, bind_tools(), agent executors. Abstracts provider differences.

• Instructor - Pydantic-based structured output library. Works with function calling across providers.

• Pydantic AI - Type-safe agent framework with validated tool definitions.

Related Technologies

• Tool Calling - The broader concept of LLMs invoking external capabilities. Function calling is the structured implementation.

• Structured Outputs - Schema-enforced LLM responses. Complementary to function calling for output formatting.

• Prompt Engineering - Tool descriptions and system prompts are prompt engineering applied to function calling.

• AI Agents - Agents are function calling wrapped in planning, memory, and orchestration loops.

• Model Context Protocol - Standardized protocol for connecting LLMs to external tools and data sources.

Learning Path

Prerequisites: Prompt Engineering · Large Language Models

Next topics: Tool Calling · Structured Outputs · AI Agents

Estimated time: 45 min · Difficulty: Intermediate

FAQs

What is the difference between function calling and tool calling?

Function calling is the technical mechanism - the model outputs structured JSON specifying a function and arguments. Tool calling is the broader concept of LLMs invoking external capabilities, which may use function calling, MCP, or other protocols.

Does the LLM execute the function?

No. The model generates a structured request. Your application code receives it, validates arguments, executes the function, and returns the result. The model never directly accesses your systems.

How many tools can I provide?

Practical limit is 10–20 well-described tools. Beyond that, selection accuracy degrades. For larger tool sets, use hierarchical selection or dynamic tool loading based on conversation context.

What happens if the model generates invalid arguments?

Your validation layer catches it. Return an error message to the model via the tool result - it will often self-correct on the next iteration. Log invalid arguments for monitoring.

Can I force the model to use a specific tool?

Yes. Set tool_choice={"type": "function", "function": {"name": "my_function"}} to force a specific tool, or tool_choice="required" to require any tool call.

How do I handle long-running tool executions?

Return immediately with a status and use async patterns. For operations taking >5s, consider returning a job ID and having the model inform the user to check back.

Should I use function calling or structured outputs?

Use function calling when the model needs to take action (query database, call API). Use structured outputs when you need a specific response format (JSON extraction, classification). They complement each other.

How do I test function calling?

Create test cases with expected tool calls: given this user message, the model should call this function with these arguments. Run against your golden set and measure selection accuracy and argument correctness.

What is tool_choice: "none"?

It disables tool calling for that request - the model responds with text only. Useful when you want to prevent tool use for certain conversation states or user types.

How does function calling work with streaming?

Tool calls are returned in the final message, not streamed token by token. You can stream the text portions while waiting for tool call completion. Some providers stream tool call arguments as they are generated.

References

• Hugging Face Transformers Documentation
• LangChain Documentation
• OpenAI API Documentation

Further Reading

• Anthropic Documentation
• Google AI for Developers
• LlamaIndex Documentation
• Pinecone Learning Center

Summary

• Function calling lets LLMs output structured tool invocations instead of free text - your code executes them. - Define tools with clear JSON Schema descriptions; the model uses descriptions to select and parameterize tools. - Always validate arguments, check permissions, and limit iteration count before executing. - The execution loop (call → execute → return → repeat) is the foundation of AI agents.

• Treat tool calls as untrusted API requests - validate, authorize, and sanitize everything. - Monitor tool selection accuracy, execution success rate, and loop iteration counts in production.