Durable Execution Guide

TL;DR

• Durable execution means agent runs survive process death - state is checkpointed so work resumes from the last safe step, not from scratch.

• Checkpointing + idempotent side effects are the core pair: persistence without idempotency causes duplicate refunds and double emails on retry.

• Long-running agents need this by default - human approvals, slow tools, rate limits, and multi-hour research all outlive a single request.

• Retries must be scoped - retry LLM/tool failures with backoff; do not blindly replay irreversible writes.

• Pick the durability layer intentionally - LangGraph checkpointers for agent graphs, Temporal/workflow engines for long-lived business processes, or both composed.

Why This Matters

Agent demos run in one process for thirty seconds. Production agents wait for humans, hit API rate limits, deploy across rolling restarts, and run for minutes to days. An in-memory for step in range(n) loop loses everything when the worker dies mid-flight.

Durable execution is the difference between "we have an agent prototype" and "we can operate agents like any other workflow." It also unlocks human-in-the-loop: approvals only work if the run can suspend for hours and resume with the same context.

Cost is another driver. Replaying five completed tool calls and three LLM turns after a crash wastes money and can re-trigger side effects. Checkpoints let you pay for the remaining work only.

The Problem Durable Execution Solves

Without durability, multi-step agents fail when:

1. The worker process restarts - deploys, OOM kills, spot instance preemption.

2. A step waits longer than the HTTP timeout - human approval, batch job, slow vendor API.

3. A transient error hits mid-run - 429 rate limit, brief DB blip - and naive retry restarts the whole task.

4. You need auditability - regulators and incident response ask "what state were we in when it failed?"

5. Multiple workers must coordinate - only one should resume a given thread_id / workflow ID.

These are classic distributed-systems problems. Agents amplify them because each step is expensive (LLM tokens) and sometimes irreversible (tools with side effects). Durable execution applies workflow-engine ideas - checkpoints, signals, retries - to agent graphs.

How We Got Here

Agent durability reused decades of workflow engineering, then specialized for non-deterministic LLM steps:

Diagram: Evolution of durable execution

flowchart LR
    A[Cron / scripts] --> B[Job queues]
    B --> C[Saga / BPMN engines]
    C --> D[Temporal-style durable workflows]
    D --> E[Agent checkpointers]
    E --> F[HITL + multi-day runs]

Agent checkpointers reuse workflow-engine ideas so LLM steps survive crashes and human waits.

Anthropic — Building effective agents stresses reliable orchestration over unbounded autonomy; durability is how that reliability survives deploys and human waits.

What Is Durable Execution?

Durable execution is a runtime property: a workflow's progress is persisted such that execution can continue after failures, pauses, or migrations without losing completed steps.

For AI agents, that usually means:

Durable execution is related to but not identical to agent memory. Memory stores knowledge across sessions; durability stores execution progress within a run (and sometimes across). You often need both.

Engineering Insight

💡 Key Idea - Durability is about the control plane of the agent (where am I in the graph?), not the intelligence of the model. A smarter model does not survive a crashed worker; a checkpoint does.

How Durable Execution Works

At a high level, the orchestrator wraps each step:

1. Load latest checkpoint for run_id.
2. Execute the next node (LLM call, tool, router).
3. Persist a new checkpoint (atomically with step completion metadata).
4. On failure, decide: retry step, compensate, or wait for signal.

Diagram: Interaction sequence

sequenceDiagram
    participant Client
    participant Worker
    participant Checkpointer
    participant Tool

    Client->>Worker: start / resume(run_id)
    Worker->>Checkpointer: load checkpoint
    Checkpointer-->>Worker: state @ step N
    Worker->>Worker: LLM / decide
    Worker->>Tool: execute (idempotency key)
    Tool-->>Worker: result
    Worker->>Checkpointer: save checkpoint N+1
    Note over Worker,Checkpointer: Process crash here is safe
    Client->>Worker: resume(run_id)
    Worker->>Checkpointer: load checkpoint N+1
    Worker->>Worker: continue from N+1

Sequence of messages and tool calls for this pattern.

Run lifecycle

Diagram: State lifecycle

stateDiagram-v2
    [*] --> Created
    Created --> Running: start
    Running --> Running: step + checkpoint
    Running --> Waiting: HITL / signal / timer
    Waiting --> Running: resume / signal
    Running --> Retrying: transient failure
    Retrying --> Running: backoff ok
    Retrying --> Failed: exhausted
    Running --> Completed: terminal
    Running --> Failed: poison / bug
    Completed --> [*]
    Failed --> [*]

Valid states and transitions for this control-plane pattern.

Lifecycle notes: Waiting must outlive the worker process. Retries apply to the current step with idempotency - not a full replay of completed tools. Graph version checks prevent new deploys from resuming into missing nodes.

Checkpoint contents

A minimal agent checkpoint includes:

• Message / event history (or pointers to blob storage for large payloads)
• Graph cursor (current node, pending edges)
• Structured state fields (order_id, plan, budgets)
• Interrupt payloads awaiting HITL
• Version of graph definition (so code deploys do not silently mismatch state)

Retry policies

Architecture

Diagram: Durable agent run architecture

flowchart TB
    Client[Client / queue] --> Worker[Stateless worker]
    Worker --> Graph[Agent graph]
    Worker --> Runtime[Durable runtime]
    Runtime --> Store[(Checkpoint store)]
    Graph --> LLM[LLM calls]
    Graph --> Tools[Tool adapters]
    Tools --> Ledger[Idempotency ledger]
    Ledger --> Ext[External APIs]
    Runtime --> Lease[Run lease / lock]
    Worker --> Obs[Traces / metrics]

Major components and how control or data moves between them.

Two common compositions:

1. Agent-native durability - LangGraph + Postgres checkpointer. Best when the graph is the product.
2. Workflow engine + LLM steps - Temporal workflow calls activities that invoke the LLM/tools. Best when the business process already lives in Temporal and agents are activities inside it.

Best Practice

✅ Best Practices - Start with a Postgres-backed checkpointer and idempotent write tools. Add a full workflow engine when you need calendars, multi-day SLAs, or complex compensation sagas.

Step-by-Step Flow

Research agent task: "Compile a competitive brief on three vendors; wait for PM approval; then email the doc."

1. Create run - Allocate run_id, persist initial state {goal, vendors[]}.

2. Research steps - For each vendor, call search/browse tools; checkpoint after each vendor so a crash mid-loop does not discard prior research.

3. Draft - LLM synthesizes brief; checkpoint stores draft blob reference (not a 50KB inline row if avoidable).

4. Interrupt for approval - HITL gate; worker releases lease. State remains durable.

5. Hours later - PM approves via UI; API signals resume with decision.

6. Worker picks up run - Loads checkpoint; skips research (already done); proceeds to email activity.

7. Email with idempotency key - email:{run_id}:brief so a double resume cannot spam.

8. Terminal checkpoint - Mark run completed; retain for audit retention policy.

9. On deploy mid-run - New workers resume old run_ids; graph version check ensures compatibility or migrates state.

Real Production Example

A compact pattern: checkpointed loop with idempotent tools and resume after failure.

from __future__ import annotations

import hashlib
import json
import time
from dataclasses import dataclass, field
from typing import Any, Callable

@dataclass
class Checkpoint:
    run_id: str
    step: int
    state: dict[str, Any]
    status: str  # "running" | "waiting" | "completed" | "failed"

class CheckpointStore:
    """Replace with Postgres in production."""

    def __init__(self):
        self._data: dict[str, Checkpoint] = {}

    def load(self, run_id: str) -> Checkpoint | None:
        return self._data.get(run_id)

    def save(self, cp: Checkpoint) -> None:
        self._data[run_id := cp.run_id] = cp

def idempotency_key(run_id: str, tool: str, args: dict) -> str:
    payload = json.dumps({"run_id": run_id, "tool": tool, "args": args}, sort_keys=True)
    return hashlib.sha256(payload.encode()).hexdigest()[:32]

@dataclass
class DurableAgent:
    store: CheckpointStore
    llm: Callable[[dict], dict]
    tools: dict[str, Callable[..., str]]
    max_steps: int = 12

    def start(self, run_id: str, goal: str) -> Checkpoint:
        cp = Checkpoint(run_id=run_id, step=0, state={"goal": goal, "messages": [], "results": []}, status="running")
        self.store.save(cp)
        return self._run(cp)

    def resume(self, run_id: str, signal: dict | None = None) -> Checkpoint:
        cp = self.store.load(run_id)
        if cp is None:
            raise KeyError(f"unknown run {run_id}")
        if signal:
            cp.state["signal"] = signal
            cp.status = "running"
        return self._run(cp)

    def _run(self, cp: Checkpoint) -> Checkpoint:
        while cp.status == "running" and cp.step < self.max_steps:
            action = self.llm(cp.state)

            if action["type"] == "wait_for_human":
                cp.state["interrupt"] = action["payload"]
                cp.status = "waiting"
                self.store.save(cp)
                return cp

            if action["type"] == "final":
                cp.state["answer"] = action["content"]
                cp.status = "completed"
                self.store.save(cp)
                return cp

            if action["type"] == "tool":
                key = idempotency_key(cp.run_id, action["name"], action["args"])
                result = self._exec_tool(action["name"], action["args"], key)
                cp.state["results"].append({"tool": action["name"], "result": result, "key": key})
                cp.state["messages"].append({"role": "tool", "content": result})
                cp.step += 1
                self.store.save(cp)  # durable boundary after each tool
                continue

            cp.status = "failed"
            cp.state["error"] = f"unknown action {action}"
            self.store.save(cp)
            return cp

        if cp.step >= self.max_steps and cp.status == "running":
            cp.status = "failed"
            cp.state["error"] = "max_steps exceeded"
            self.store.save(cp)
        return cp

    def _exec_tool(self, name: str, args: dict, key: str) -> str:
        # Production: consult a side-effect ledger keyed by `key` before calling out.
        return self.tools[name](**args, idempotency_key=key)

# --- LangGraph-shaped sketch (production) ---
# from langgraph.checkpoint.postgres import PostgresSaver
# with PostgresSaver.from_conn_string(DATABASE_URL) as checkpointer:
#     app = graph.compile(checkpointer=checkpointer)
#     app.invoke(input, {"configurable": {"thread_id": run_id}})
#     # later / after crash:
#     app.invoke(None, {"configurable": {"thread_id": run_id}})  # or Command(resume=...)

The important boundaries: save after each side-effecting step, suspend cleanly for humans, and key writes so resume cannot double-apply.

Design Decisions

Comparisons

Durable Execution vs Retries

Retries are a policy inside durability. Exponential backoff without checkpoints still loses the run on deploy. See Temporal durable execution for the classic workflow framing.

Durable Execution vs Agent Memory

You need both for production agents: agent memory for knowledge, durability for control-plane progress. Confusing them is a common design bug.

Durable Execution vs HITL

Decision tree: do you need durability?

Decision tree: When to add durable execution

flowchart TD
    A[Multi-step agent or workflow?] -->|No| B[Stateless request is enough]
    A -->|Yes| C{Can the run outlive one process?}
    C -->|No| B
    C -->|Yes| D[Checkpoint after safe steps]
    D --> E{Human approval or long waits?}
    E -->|Yes| F[Durable status + resume API]
    E -->|No| G[Still checkpoint for deploys/retries]
    F --> H[Pair with idempotent side effects]
    G --> H

If a deploy mid-run would force a full replay, you need checkpoints - not just retries.

Human-in-the-loop is the product feature; durable execution is the infrastructure that makes interrupt/resume real - documented clearly in LangGraph HITL + persistence.

⚠ Common Mistakes

1. In-memory state in production - Works until the first deploy during a run.

2. Checkpointing without idempotent tools - Resume duplicates payments and emails.

3. Giant checkpoints - Stuffing full scraped HTML into Postgres rows blows latency and storage. Store references.

4. Retrying non-idempotent activities blindly - Distinguish transient infra failures from "already succeeded, response lost."

5. Ignoring graph version skew - Deploying new node names while old runs resume into missing nodes.

6. Confusing memory with durability - Vector memory does not resume a half-finished refund workflow.

7. One global worker without leasing - Two workers resume the same run_id and race side effects.

Where It Breaks Down

• Ultra-low-latency single-shot tools - Durability overhead is unnecessary for one LLM call with no side effects.

• Non-deterministic replay assumptions - Some workflow engines replay history deterministically; LLM calls are not deterministic. Prefer checkpoint-and-continue over full event replay for model steps, or record model outputs as history events.

• Huge fan-out - Millions of concurrent short runs may need a lighter store or batching strategy.

• Poison messages - A bad state that crashes every resume needs a dead-letter path and manual repair tools.

• Cross-system transactions - Durability of the agent does not make a multi-API business operation atomic. Use sagas/compensations.

When NOT to Use Durable Execution

Skip a full durability layer when:

1. Single-shot, no side effects - one LLM call or one read-only tool; crash cost is a cheap retry.
2. Runs always finish inside one request - no HITL, no multi-minute tools, no deploy mid-flight risk you care about.
3. You cannot make writes idempotent - checkpoints without idempotency keys make duplicates more likely on resume.
4. Checkpoint payloads would be huge and unscoped - scraping full HTML into every step; redesign blobs/pointers first.
5. A parent workflow engine already owns the saga - e.g. Temporal activities wrap the LLM; do not double-persist conflicting cursors.
6. Latency budget is tighter than checkpoint I/O and you have no multi-step risk - rare; usually LLM time dominates.

Otherwise prefer agent-native checkpointers (LangGraph) or a workflow engine - compare orchestration options in Best AI Agent Frameworks.

Running in Production

Best Practice

✅ Best Practices - Persist after every side effect, lease runs to one worker, version your graphs, keep checkpoints lean, and wrap external writes with idempotency keys.

Important

At-least-once delivery is the realistic default. Design every tool adapter as if the previous attempt might have succeeded. Query provider state or use idempotency keys before creating a new side effect.

Common Mistake

Using MemorySaver (or equivalent) in production "temporarily." It will be forgotten until an outage deletes every in-flight agent. Configure Postgres/Redis checkpointers before the first real user traffic.

Related Guides

• LangGraph - Native checkpointers (PostgresSaver, SqliteSaver) and thread-based resume; pairs cleanly with HITL interrupts.
• Temporal - General-purpose durable workflows; use when agents are steps inside longer business processes.
• CrewAI - Crew/flow persistence patterns for multi-step agent teams.
• AutoGen - Conversational multi-agent runs that need external persistence for production.
• OpenAI Agents SDK - Session/run persistence hooks depending on deployment model.

Compare agent orchestration stacks:

• Best AI Agent Frameworks

• LangGraph vs CrewAI

• Human-in-the-Loop - The most common reason to suspend and resume durable runs.

• AI Agents - The loops you make durable.

• Workflows vs Agents - When a deterministic workflow engine should own durability.

• Agent Memory - Cross-session knowledge vs. in-run checkpoints.

• Multi-Agent Systems - Fan-out durability and shared state.

• Multi-Agent Handoffs - Handoffs must carry durable run identity and state.

If you understood this topic, read next:

Diagram: Learning path for durable agents

flowchart LR
    A[Durable] --> B[HITL]
    B --> C[Handoffs]
    C --> D[Memory]
    D --> E[Workflows]

Prerequisites: AI Agents · Workflows vs Agents

Next topics: Human-in-the-Loop · Agent Memory · Multi-Agent Handoffs

Estimated time: 55 min · Difficulty: Intermediate

Key Takeaways

• Durable execution persists control-plane progress so crashes, deploys, and human waits do not wipe completed steps.
• Checkpoint after side effects; pair with idempotent tools - persistence alone is not safe.
• Retries are a policy; they do not replace checkpoints. Agent memory is knowledge, not a run cursor.
• HITL depends on durability: approvals arrive minutes later to a resumed run_id.
• Choose agent-native checkpointers (LangGraph) or engines like Temporal based on who owns the business saga.
• Version graphs, lease runs to one worker, and chaos-test kill/resume for duplicate side effects.
• Compare stacks via Best AI Agent Frameworks and LangGraph vs CrewAI.

FAQs

Is durable execution the same as storing chat history?

No. Chat history is one field inside state. Durability also tracks graph position, interrupts, retries, and run lifecycle so execution can continue correctly.

Do I need Temporal if I use LangGraph checkpointers?

Not always. LangGraph persistence covers many agent apps. Choose Temporal when you already standardize on it for business workflows, need complex timers/sagas, or run multi-day processes with strict ops tooling.

What should I checkpoint after?

At minimum after every side-effecting tool call. After every node is safer for long reasoning chains.

How do I prevent duplicate side effects on resume?

Idempotency keys, upsert semantics, and status checks against the external system before creating new resources.

Can I change my graph code while runs are in flight?

Yes with care: version graphs, avoid deleting nodes that in-flight runs still need, and migrate state when schemas change.

How long should checkpoints be retained?

Match audit and debugging needs - often 30–90 days for production agents. Separate hot (active runs) from cold (completed) storage if volume grows.

What about secrets in checkpoints?

Do not persist raw API keys or credentials. Redact tool args that contain secrets; store references to a secret manager.

How does this relate to serverless timeouts?

Durable execution is how you survive 15–60s function limits: each invocation loads state, runs one or few steps, saves, exits; a queue continues the run.

Should LLM calls be deterministic for replay?

Prefer recording the LLM output in the checkpoint and continuing forward, rather than re-calling the model during "replay" of history.

What metrics indicate durability problems?

Spike in duplicate side effects, rising average resumes per run, checkpoint write errors, and stuck waiting runs beyond SLA.

Is Redis durable enough?

With AOF/replication and short-lived runs, often yes. For financial workflows and long HITL waits, prefer Postgres or a workflow engine's store.

How do multi-agent systems checkpoint?

Either one shared run state with sub-agent spans, or child workflows with parent orchestration. Always propagate a correlation run_id.

Does durability make agents slower?

Slightly per step; massively faster/cheaper after failures because you do not redo completed work.

What's the first production checklist item?

Replace in-memory savers, add thread_id/run_id to every invoke, and wrap the top three write tools with idempotency.

References

• LangGraph Persistence
• Temporal Durable Execution
• Anthropic - Building effective agents

Further Reading

• LangGraph Checkpointing docs
• OpenAI Agents SDK
• CrewAI Flows
• Best AI Agent Frameworks ranking