Memory Guide

TL;DR

• Agent memory spans three timescales: short-term (in-context conversation), long-term (persistent semantic recall), and episodic (structured task/event history).

• Context windows are not memory - they're a cache with a size limit. Production agents externalize memory to vector stores, databases, and summarization pipelines.

• Retrieval beats accumulation - storing everything in the prompt doesn't scale.

Retrieve relevant memories on demand, like RAG but for agent experience.

• Memory writes need curation - indiscriminate storage creates noisy retrieval. Decide what to remember, when to remember, and when to forget.

• Memory is a security surface - stored interactions can leak PII across users if tenant isolation isn't enforced at the memory layer.

Why This Matters

A support agent that forgets a customer's issue between messages is useless. A coding agent that doesn't remember the file structure from five minutes ago re-reads everything and burns tokens. A personal assistant that can't recall your preferences from last week is just a chatbot with extra steps.

Memory is what makes agents stateful - able to maintain continuity across turns, sessions, and tasks. It's also one of the hardest engineering problems in agent systems because LLM context windows are finite, expensive, and degrade in quality as they fill up.

Every production agent eventually hits the same wall: "We need it to remember X." How you answer that - summarization, vector retrieval, structured state, or a combination - determines whether your agent feels intelligent or amnesiac.

The Problem Agent Memory Solves

LLMs are stateless functions. Each API call receives a context window and produces a response. Without external memory:

1. Multi-turn conversations lose early context as the window fills or gets truncated.

2. Cross-session continuity is impossible - the agent starts fresh every time.

3. Task progress isn't tracked - completed steps, intermediate results, and decisions evaporate.

4. Personalization can't accumulate - user preferences, past interactions, and learned patterns aren't retained.

5. Token costs explode - resending full history on every call is expensive and slow.

Memory systems externalize state so agents can read what they need, write what matters, and forget what's stale - without stuffing everything into the prompt.

How We Got Here

Agent memory evolved from chat history hacks into a layered persistence stack:

Diagram: Short-term vs long-term agent memory

flowchart LR
    A[Full chat replay] --> B[Truncation / sliding window]
    B --> C[Summarization buffers]
    C --> D[Vector semantic recall]
    D --> E[Episodic + checkpoint stores]
    E --> F[Curated memory + TTL]

Major components and how control or data moves between them.

Public systems that shaped the pattern include LangGraph persistence, Mem0, and Zep. The lesson: memory is a data product, not a longer prompt.

What Is Agent Memory?

Agent memory is the set of mechanisms that store, retrieve, update, and expire information an agent needs beyond the current context window. It maps loosely to human memory types:

In practice, most agent systems implement short-term (message history), working (state dict), and long-term (vector store) memory. Episodic and procedural memory are often layered on top as the system matures.

How Agent Memory Works

Write Path: What Gets Stored

Not everything should be remembered. Effective memory systems decide at write time:

def should_remember(event: AgentEvent) -> bool:
    if event.type == "tool_error":
        return True  # Learn from failures
    if event.type == "user_preference":
        return True  # Personalization
    if event.type == "routine_tool_call":
        return False  # Noise
    if event.type == "final_answer":
        return True  # Episodic record
    return False

Common write triggers:

• Explicit user instruction: "Remember that my project uses PostgreSQL"
• Agent self-reflection: Reflexion critiques stored for future attempts
• Task completion: Summary of what was done and outcome
• Periodic summarization: Compress conversation segment into a summary memory

Read Path: What Gets Retrieved

At each agent step, relevant memories are retrieved and injected into context:

def build_context(user_message, user_id, session_id):
    # Short-term: recent messages (truncated/summarized if needed)
    recent = message_store.get_recent(session_id, limit=20)

    # Long-term: semantic search over user memories
    relevant = memory_store.search(
        query=user_message,
        filter={"user_id": user_id},
        top_k=5,
    )

    # Working: current task state
    state = state_store.get(session_id)

    return compose_prompt(recent, relevant, state)

Forgetting: What Gets Expired

Memory without expiration becomes a liability - stale data, privacy risk, retrieval noise. Strategies:

• TTL - expire memories after N days unless reinforced.

• LRU eviction - keep most recently accessed memories within storage budget.

• Summarization - compress old conversation segments into summary memories, delete raw messages.

• Explicit deletion - honor user requests to forget specific information.

Memory read/write lifecycle

Diagram: Memory read/write lifecycle

flowchart TD
    U[User turn] --> W{Write filter}
    W -->|keep| S[(Stores)]
    W -->|drop| X[Discard noise]
    S --> H[Message history]
    S --> V[Vector semantic]
    S --> E[Episodic events]
    S --> C[Working checkpoint]
    U --> R[Retrieve top-k]
    R --> H
    R --> V
    R --> E
    R --> C
    R --> P[Compose prompt]
    P --> A[Agent step]

Major components and how control or data moves between them.

Architecture

Production memory architecture with three tiers:

Diagram: Layered memory architecture

flowchart TB
    Agent[Agent loop] --> Short[Short-term messages]
    Agent --> Work[Working state / checkpoint]
    Agent --> Long[Long-term semantic]
    Agent --> Epi[Episodic event log]
    Short --> Sum[Summarizer]
    Sum --> Long
    Work --> CP[(Postgres / Redis checkpointer)]
    Long --> VS[(Vector store)]
    Epi --> DB[(Event DB)]

Major components and how control or data moves between them.

Step-by-Step Flow

Scenario: Returning user asks "Can you continue the migration we started yesterday?"

1. Session lookup - Agent receives user_id and new session_id. Checks for active task state in state store.

2. Episodic query - Query episodic DB: events WHERE user_id=X AND tag='migration' ORDER BY timestamp DESC LIMIT 5. Finds yesterday's session with task summary.

3. Long-term retrieval - Semantic search over user memories with query "database migration" returns: "User's project uses PostgreSQL 15", "Migration target is Aurora", "Completed schema backup step".

4. Context assembly - System prompt + episodic summary + retrieved memories + new user message. Recent messages from current session (empty - new session).

5. Agent reasoning - Agent recognizes migration context, loads working state from checkpoint (if persisted), identifies next step: "execute schema migration on staging."

6. Execution - Agent proceeds with full context despite new session ID.

7. Memory write - On task progress, update working state checkpoint. Write episodic event: "Resumed migration, completed staging schema sync."

8. Summarization - If conversation grows long, summarize first 15 messages into long-term memory, truncate raw history.

Real Production Example

Agent memory with LangGraph checkpointing and a vector memory store:

from langgraph.checkpoint.postgres import PostgresSaver
from langgraph.graph import StateGraph
from typing_extensions import TypedDict
from typing import Annotated
from langgraph.graph.message import add_messages
from langchain_openai import OpenAIEmbeddings, ChatOpenAI
import uuid

# --- Long-term memory store (simplified) ---
class MemoryStore:
    def __init__(self, vectorstore):
        self.vs = vectorstore
        self.embeddings = OpenAIEmbeddings(model="text-embedding-3-small")

    def add(self, user_id: str, content: str, metadata: dict = None):
        self.vs.add_texts(
            texts=[content],
            metadatas=[{"user_id": user_id, **(metadata or {})}],
            ids=[str(uuid.uuid4())],
        )

    def search(self, user_id: str, query: str, top_k: int = 5) -> list[str]:
        results = self.vs.similarity_search(
            query,
            k=top_k,
            filter={"user_id": user_id},
        )
        return [doc.page_content for doc in results]

memory = MemoryStore(vectorstore)

# --- Agent state with memory injection ---
class AgentState(TypedDict):
    messages: Annotated[list, add_messages]
    user_id: str
    retrieved_memories: list[str]

llm = ChatOpenAI(model="gpt-4o", temperature=0)

def retrieve_memories(state: AgentState):
    last_user_msg = next(m for m in reversed(state["messages"]) if m.type == "human")
    memories = memory.search(state["user_id"], last_user_msg.content)
    return {"retrieved_memories": memories}

def agent_with_memory(state: AgentState):
    memory_context = "\n".join(f"- {m}" for m in state["retrieved_memories"])
    system_msg = f"""You are a helpful assistant with access to the user's history.

Relevant memories:
{memory_context or "No prior memories found."}
"""
    messages = [{"role": "system", "content": system_msg}] + state["messages"]
    response = llm.invoke(messages)
    return {"messages": [response]}

def save_memories(state: AgentState):
    last_exchange = state["messages"][-2:]
    summary = llm.invoke([
        {"role": "system", "content": "Extract key facts worth remembering.

Return as bullet points or NONE."},
        {"role": "user", "content": str(last_exchange)},
    ])
    if "NONE" not in summary.content:
        for line in summary.content.strip().split("\n"):
            if line.strip():
                memory.add(state["user_id"], line.strip(), {"source": "conversation"})
    return {}

# --- Graph with checkpointing ---
graph = StateGraph(AgentState)
graph.add_node("retrieve", retrieve_memories)
graph.add_node("agent", agent_with_memory)
graph.add_node("save", save_memories)
graph.set_entry_point("retrieve")
graph.add_edge("retrieve", "agent")
graph.add_edge("agent", "save")

checkpointer = PostgresSaver.from_conn_string("postgresql://...")
agent = graph.compile(checkpointer=checkpointer)

# Resume across sessions with same thread_id
config = {"configurable": {"thread_id": f"user-{user_id}"}}
result = agent.invoke({"messages": [("user", "Continue the migration")], "user_id": user_id}, config)

Key patterns: retrieve before reasoning, save after response, checkpoint for session continuity, tenant-scoped vector search.

Design Decisions

Comparisons

Memory vs durable execution checkpoints

Checkpoints are working-memory snapshots for reliability. Long-term memory is a knowledge product. Production AI agents usually need both.

Memory vs context window

Episodic vs semantic memory

Decision tree: which memory layer?

Decision tree: Choosing an agent memory layer

flowchart TD
    A[Need state across steps in one run?] -->|Yes| B[Working state / durable checkpoint]
    A -->|No| C[Need facts across sessions?]
    C -->|No| D[Message history may be enough]
    C -->|Yes| E{Exact events or soft preferences?}
    E -->|Events / audit| F[Episodic store]
    E -->|Preferences / semantics| G[Vector semantic memory]
    B --> H[Pair with durable execution]
    F --> I[Tenant filters + TTL]
    G --> I

Do not put order IDs in a vector store - use checkpoints for control-plane state and retrieval memory for knowledge.

Common Mistakes

1. Treating the context window as unlimited memory - Quality degrades as context fills ("lost in the middle" effect). Externalize early.

2. Storing everything - Retrieval returns noise. Implement write filters and importance scoring.

3. No tenant isolation - User A's memories surfacing for User B is a privacy incident. Filter vector searches by user/tenant ID.

4. Never forgetting - Stale memories contradict current state. Implement TTL and explicit deletion.

5. Retrieving too many memories - 20 memory chunks in context adds noise. Start with top_k=3–5.

6. No memory evaluation - You can't improve what you don't measure. Track: retrieval relevance, memory hit rate, user corrections ("that's outdated").

7. Ignoring checkpointing - Without persistent state, agents can't resume interrupted tasks or survive restarts.

Where It Breaks Down

• Conflicting memories - "User prefers email" from 2023 vs. "User prefers Slack" from 2025. Need recency weighting or explicit superseding.

• Memory injection attacks - Adversarial content stored in long-term memory can influence future sessions (persistent prompt injection). Sanitize writes.

• Summarization loss - Aggressive summarization drops critical details. Keep raw messages for recent history; summarize only older segments.

• Cross-session entity resolution - "Continue the project" - which project? Need disambiguation or episodic metadata.

• Scale - Millions of memories per user degrade retrieval speed. Partition by time, topic, or use hierarchical retrieval.

When NOT to Use Agent Memory

Skip a dedicated memory layer when:

1. Single-turn or short sessions - FAQ bots and one-shot tools don't need cross-session recall.
2. All state fits a small deterministic store - order IDs and ticket status belong in your app DB, not a vector memory.
3. You cannot enforce tenant isolation - shared memory without RBAC is a privacy incident waiting to happen.
4. Write volume would be unfiltered noise - if you cannot curate what to remember, retrieval will degrade.
5. Compliance forbids retention - some flows must be ephemeral; use TTL-zero or no long-term store.

Prefer context windows + summarization for short chats, durable execution for resume-only needs, and RAG for document knowledge (not user experience).

Running in Production

Best Practice

✅ Best Practices - Curate writes, retrieve narrowly, isolate tenants, expire stale memories, and measure recall across sessions before widening retention.

Important

Memory is a persistence layer for user data. Apply the same security, privacy, and compliance controls as any database storing user information.

Related Guides

• LangGraph checkpointing: Postgres, SQLite, Redis backends for session state persistence - compare frameworks in Best AI Agent Frameworks.

• Mem0: Purpose-built agent memory layer with automatic extraction and deduplication.

• Zep: Temporal knowledge graph for agent memory with fact invalidation.

• LangChain Memory modules: ConversationBufferMemory, ConversationSummaryMemory, vector store memory.

• Vector stores: Pinecone, Weaviate, pgvector - same infrastructure as RAG, different content. See Best Vector Databases.

• Comparisons: Pinecone vs Weaviate · LangGraph vs CrewAI (checkpointing / orchestration tradeoffs)

• Context Windows: The finite resource memory systems work around.

• Embeddings: Foundation of semantic long-term memory retrieval.

• RAG: Memory retrieval is structurally similar to RAG - search before generate.

• Durable Execution: Checkpoint/resume for long-running agent runs.

• AI Agents: Memory enables stateful agent behavior across turns and sessions.

• Agent Architectures: Reflexion and replanning depend on memory of past attempts.

• Agentic RAG: Agents that retrieve knowledge adaptively - memory and RAG converge.

• Human-in-the-Loop: Approvals need durable state while waiting for humans.

• AI Security: Memory stores are an attack surface for persistent prompt injection.

If you understood this topic, read next:

Diagram: Learning path for agent memory

flowchart LR
    A[Agents] --> B[Memory]
    B --> C[Durable]
    C --> D[HITL]
    D --> E[Agentic RAG]
    E --> F[Handoffs]

Prerequisites: AI Agents · Context Windows · Embeddings

Next topics: Durable Execution · Agentic RAG · Agent Architectures

Estimated time: 45 min · Difficulty: Intermediate

Key Takeaways

• Agent memory spans short-term (messages), working (state/checkpoints), and long-term (semantic/episodic) timescales.
• Context windows are not memory - externalize state before you hit limits or quality cliffs.
• Be selective about writes; unfiltered memory degrades retrieval. Retrieve top_k=3–5 on demand.
• Checkpoints resume runs; semantic/episodic stores recall knowledge - see Durable Execution.
• Enforce tenant isolation - memory is user data with privacy and security requirements.
• Evaluate with cross-turn and cross-session recall tests, not just single-turn accuracy.
• Combine memory with RAG / Agentic RAG when agents need both experience and documents.

FAQs

What's the difference between agent memory and RAG?

RAG retrieves external documents (knowledge base). Agent memory retrieves the agent's own experience (past interactions, preferences, task history). Same retrieval mechanism, different content source. Many systems combine both.

How much conversation history should I keep in context?

Keep the most recent 10–20 messages raw. Summarize anything older. Exact count depends on message length and task complexity - monitor context utilization.

Should I use a separate vector store for memory vs. RAG?

For small systems, one store with metadata filters works. At scale, separate stores prevent retrieval noise - a "user preference" shouldn't compete with "product documentation" in the same search.

How does LangGraph checkpointing relate to memory?

Checkpointing persists agent state (messages, plan, variables) for crash recovery and session resume. It's working memory, not long-term semantic memory - but it's essential for production reliability.

What is episodic memory in agents?

A timestamped log of agent events: tasks attempted, outcomes, tools used, errors encountered. Queryable by time, user, or tag. Useful for audit, analytics, and "what happened last time" queries.

How do I handle "forget that" requests?

Delete specific memories from the vector store and episodic DB by ID or semantic search match. Confirm deletion to the user. Required for GDPR and similar regulations.

Can agents share memory across users?

Only with explicit design - team knowledge bases, shared project context. Default to per-user isolation. Shared memory needs RBAC to prevent cross-user leakage.

How do I prevent memory from growing unbounded?

TTL on memories, LRU eviction, periodic summarization and compaction, per-user storage quotas, and importance scoring that drops low-value memories.

What's the "lost in the middle" problem?

LLMs attend less to information in the middle of long contexts. Relevant memories placed mid-prompt may be ignored. Put critical memories near the start or end of context.

Should memory writes be synchronous or async?

Async for non-critical writes (conversation summaries). Synchronous when the next step depends on the write (checkpoint before tool execution). Never block user response on memory writes.

How do I evaluate memory quality?

Test scenarios: state a preference, continue 20 turns later, verify recall. Cross-session tests. Adversarial tests: verify User A can't retrieve User B's memories.

What's Mem0 and when should I use it?

Mem0 is a memory layer that automatically extracts, deduplicates, and manages agent memories. Use it when you don't want to build write filtering and retrieval logic from scratch.

References

• LangGraph Persistence
• Mem0 Documentation
• Zep Documentation

Further Reading

• Anthropic — Building effective agents
• Lost in the Middle (Liu et al., 2023)
• LangGraph Documentation