Memory Guide

TL;DR

• Agent memory spans three timescales: short-term (in-context conversation), long-term (persistent semantic recall), and episodic (structured task/event history).

• Context windows are not memory - they're a cache with a size limit. Production agents externalize memory to vector stores, databases, and summarization pipelines.

• Retrieval beats accumulation - storing everything in the prompt doesn't scale.

Retrieve relevant memories on demand, like RAG but for agent experience.

• Memory writes need curation - indiscriminate storage creates noisy retrieval. Decide what to remember, when to remember, and when to forget.

• Memory is a security surface - stored interactions can leak PII across users if tenant isolation isn't enforced at the memory layer.

Why This Matters

A support agent that forgets a customer's issue between messages is useless. A coding agent that doesn't remember the file structure from five minutes ago re-reads everything and burns tokens. A personal assistant that can't recall your preferences from last week is just a chatbot with extra steps.

Memory is what makes agents stateful - able to maintain continuity across turns, sessions, and tasks. It's also one of the hardest engineering problems in agent systems because LLM context windows are finite, expensive, and degrade in quality as they fill up.

Every production agent eventually hits the same wall: "We need it to remember X." How you answer that - summarization, vector retrieval, structured state, or a combination - determines whether your agent feels intelligent or amnesiac.

The Problem Agent Memory Solves

LLMs are stateless functions. Each API call receives a context window and produces a response. Without external memory:

1. Multi-turn conversations lose early context as the window fills or gets truncated.

2. Cross-session continuity is impossible - the agent starts fresh every time.

3. Task progress isn't tracked - completed steps, intermediate results, and decisions evaporate.

4. Personalization can't accumulate - user preferences, past interactions, and learned patterns aren't retained.

5. Token costs explode - resending full history on every call is expensive and slow.

Memory systems externalize state so agents can read what they need, write what matters, and forget what's stale - without stuffing everything into the prompt.

What Is Agent Memory?

Agent memory is the set of mechanisms that store, retrieve, update, and expire information an agent needs beyond the current context window. It maps loosely to human memory types:

In practice, most agent systems implement short-term (message history), working (state dict), and long-term (vector store) memory. Episodic and procedural memory are often layered on top as the system matures.

How Agent Memory Works

Write Path: What Gets Stored

Not everything should be remembered. Effective memory systems decide at write time:

def should_remember(event: AgentEvent) -> bool:
    if event.type == "tool_error":
        return True  # Learn from failures
    if event.type == "user_preference":
        return True  # Personalization
    if event.type == "routine_tool_call":
        return False  # Noise
    if event.type == "final_answer":
        return True  # Episodic record
    return False

Common write triggers:

• Explicit user instruction: "Remember that my project uses PostgreSQL"
• Agent self-reflection: Reflexion critiques stored for future attempts
• Task completion: Summary of what was done and outcome
• Periodic summarization: Compress conversation segment into a summary memory

Read Path: What Gets Retrieved

At each agent step, relevant memories are retrieved and injected into context:

def build_context(user_message, user_id, session_id):
    # Short-term: recent messages (truncated/summarized if needed)
    recent = message_store.get_recent(session_id, limit=20)

    # Long-term: semantic search over user memories
    relevant = memory_store.search(
        query=user_message,
        filter={"user_id": user_id},
        top_k=5,
    )

    # Working: current task state
    state = state_store.get(session_id)

    return compose_prompt(recent, relevant, state)

Forgetting: What Gets Expired

Memory without expiration becomes a liability - stale data, privacy risk, retrieval noise. Strategies:

• TTL - expire memories after N days unless reinforced.

• LRU eviction - keep most recently accessed memories within storage budget.

• Summarization - compress old conversation segments into summary memories, delete raw messages.

• Explicit deletion - honor user requests to forget specific information.

Agent systems extend LLMs with tools, memory, and planning loops so they can take actions in external environments rather than only emit text.

Architecture

Production memory architecture with three tiers:

Multi-agent systems assign specialized roles to multiple LLM agents that communicate, delegate, and coordinate to solve complex tasks.

Source: Research paper

Message History - Append-only log of conversation turns. Truncated or summarized when approaching context limits.

Summarizer - LLM call that compresses older messages into a concise summary memory, preserving key facts and decisions.

State Store - Structured key-value or document store for task progress, plan state, tool outputs. LangGraph checkpointing uses this pattern.

Vector Store - Embedding-indexed memories for semantic retrieval. Filtered by user/tenant ID.

Episodic DB - Timestamped event log for audit, analytics, and time-based queries.

Step-by-Step Flow

Scenario: Returning user asks "Can you continue the migration we started yesterday?"

1. Session lookup - Agent receives user_id and new session_id. Checks for active task state in state store.

2. Episodic query - Query episodic DB: events WHERE user_id=X AND tag='migration' ORDER BY timestamp DESC LIMIT 5. Finds yesterday's session with task summary.

3. Long-term retrieval - Semantic search over user memories with query "database migration" returns: "User's project uses PostgreSQL 15", "Migration target is Aurora", "Completed schema backup step".

4. Context assembly - System prompt + episodic summary + retrieved memories + new user message. Recent messages from current session (empty - new session).

5. Agent reasoning - Agent recognizes migration context, loads working state from checkpoint (if persisted), identifies next step: "execute schema migration on staging."

6. Execution - Agent proceeds with full context despite new session ID.

7. Memory write - On task progress, update working state checkpoint. Write episodic event: "Resumed migration, completed staging schema sync."

8. Summarization - If conversation grows long, summarize first 15 messages into long-term memory, truncate raw history.

Real Production Example

Agent memory with LangGraph checkpointing and a vector memory store:

from langgraph.checkpoint.postgres import PostgresSaver
from langgraph.graph import StateGraph
from typing_extensions import TypedDict
from typing import Annotated
from langgraph.graph.message import add_messages
from langchain_openai import OpenAIEmbeddings, ChatOpenAI
import uuid

# --- Long-term memory store (simplified) ---
class MemoryStore:
    def __init__(self, vectorstore):
        self.vs = vectorstore
        self.embeddings = OpenAIEmbeddings(model="text-embedding-3-small")

    def add(self, user_id: str, content: str, metadata: dict = None):
        self.vs.add_texts(
            texts=[content],
            metadatas=[{"user_id": user_id, **(metadata or {})}],
            ids=[str(uuid.uuid4())],
        )

    def search(self, user_id: str, query: str, top_k: int = 5) -> list[str]:
        results = self.vs.similarity_search(
            query,
            k=top_k,
            filter={"user_id": user_id},
        )
        return [doc.page_content for doc in results]

memory = MemoryStore(vectorstore)

# --- Agent state with memory injection ---
class AgentState(TypedDict):
    messages: Annotated[list, add_messages]
    user_id: str
    retrieved_memories: list[str]

llm = ChatOpenAI(model="gpt-4o", temperature=0)

def retrieve_memories(state: AgentState):
    last_user_msg = next(m for m in reversed(state["messages"]) if m.type == "human")
    memories = memory.search(state["user_id"], last_user_msg.content)
    return {"retrieved_memories": memories}

def agent_with_memory(state: AgentState):
    memory_context = "\n".join(f"- {m}" for m in state["retrieved_memories"])
    system_msg = f"""You are a helpful assistant with access to the user's history.

Relevant memories:
{memory_context or "No prior memories found."}
"""
    messages = [{"role": "system", "content": system_msg}] + state["messages"]
    response = llm.invoke(messages)
    return {"messages": [response]}

def save_memories(state: AgentState):
    last_exchange = state["messages"][-2:]
    summary = llm.invoke([
        {"role": "system", "content": "Extract key facts worth remembering.

Return as bullet points or NONE."},
        {"role": "user", "content": str(last_exchange)},
    ])
    if "NONE" not in summary.content:
        for line in summary.content.strip().split("\n"):
            if line.strip():
                memory.add(state["user_id"], line.strip(), {"source": "conversation"})
    return {}

# --- Graph with checkpointing ---
graph = StateGraph(AgentState)
graph.add_node("retrieve", retrieve_memories)
graph.add_node("agent", agent_with_memory)
graph.add_node("save", save_memories)
graph.set_entry_point("retrieve")
graph.add_edge("retrieve", "agent")
graph.add_edge("agent", "save")

checkpointer = PostgresSaver.from_conn_string("postgresql://...")
agent = graph.compile(checkpointer=checkpointer)

# Resume across sessions with same thread_id
config = {"configurable": {"thread_id": f"user-{user_id}"}}
result = agent.invoke({"messages": [("user", "Continue the migration")], "user_id": user_id}, config)

Key patterns: retrieve before reasoning, save after response, checkpoint for session continuity, tenant-scoped vector search.

Design Decisions

⚠ Common Mistakes

1. Treating the context window as unlimited memory - Quality degrades as context fills ("lost in the middle" effect). Externalize early.

2. Storing everything - Retrieval returns noise. Implement write filters and importance scoring.

3. No tenant isolation - User A's memories surfacing for User B is a privacy incident. Filter vector searches by user/tenant ID.

4. Never forgetting - Stale memories contradict current state. Implement TTL and explicit deletion.

5. Retrieving too many memories - 20 memory chunks in context adds noise. Start with top_k=3–5.

6. No memory evaluation - You can't improve what you don't measure. Track: retrieval relevance, memory hit rate, user corrections ("that's outdated").

7. Ignoring checkpointing - Without persistent state, agents can't resume interrupted tasks or survive restarts.

Where It Breaks Down

• Conflicting memories - "User prefers email" from 2023 vs. "User prefers Slack" from 2025. Need recency weighting or explicit superseding.

• Memory injection attacks - Adversarial content stored in long-term memory can influence future sessions (persistent prompt injection). Sanitize writes.

• Summarization loss - Aggressive summarization drops critical details. Keep raw messages for recent history; summarize only older segments.

• Cross-session entity resolution - "Continue the project" - which project? Need disambiguation or episodic metadata.

• Scale - Millions of memories per user degrade retrieval speed. Partition by time, topic, or use hierarchical retrieval.

Running in Production

Best Practice

✅ Best Practices - Instrument every stage, version embedding models, enforce access control at retrieval time, and evaluate on a fixed golden set before shipping changes.

Important

Memory is a persistence layer for user data. Apply the same security, privacy, and compliance controls as any database storing user information.

Ecosystem

• LangGraph checkpointing: Postgres, SQLite, Redis backends for session state persistence.

• Mem0: Purpose-built agent memory layer with automatic extraction and deduplication.

• Zep: Temporal knowledge graph for agent memory with fact invalidation.

• LangChain Memory modules: ConversationBufferMemory, ConversationSummaryMemory, vector store memory.

• Vector stores: Pinecone, Weaviate, pgvector - same infrastructure as RAG, different content.

Related Technologies

• Context Windows: The finite resource memory systems work around.

• Embeddings: Foundation of semantic long-term memory retrieval.

• RAG: Memory retrieval is structurally similar to RAG - search before generate.

• AI Agents: Memory enables stateful agent behavior across turns and sessions.

• Agent Architectures: Reflexion and replanning depend on memory of past attempts.

• Agentic RAG: Agents that retrieve knowledge adaptively - memory and RAG converge.

• AI Security: Memory stores are an attack surface for persistent prompt injection.

Learning Path

Prerequisites: AI Agents · Context Windows · Embeddings

Next topics: Agentic RAG · Agent Architectures · RAG

Estimated time: 45 min · Difficulty: Intermediate

FAQs

What's the difference between agent memory and RAG?

RAG retrieves external documents (knowledge base). Agent memory retrieves the agent's own experience (past interactions, preferences, task history). Same retrieval mechanism, different content source. Many systems combine both.

How much conversation history should I keep in context?

Keep the most recent 10–20 messages raw. Summarize anything older. Exact count depends on message length and task complexity - monitor context utilization.

Should I use a separate vector store for memory vs. RAG?

For small systems, one store with metadata filters works. At scale, separate stores prevent retrieval noise - a "user preference" shouldn't compete with "product documentation" in the same search.

How does LangGraph checkpointing relate to memory?

Checkpointing persists agent state (messages, plan, variables) for crash recovery and session resume. It's working memory, not long-term semantic memory - but it's essential for production reliability.

What is episodic memory in agents?

A timestamped log of agent events: tasks attempted, outcomes, tools used, errors encountered. Queryable by time, user, or tag. Useful for audit, analytics, and "what happened last time" queries.

How do I handle "forget that" requests?

Delete specific memories from the vector store and episodic DB by ID or semantic search match. Confirm deletion to the user. Required for GDPR and similar regulations.

Can agents share memory across users?

Only with explicit design - team knowledge bases, shared project context. Default to per-user isolation. Shared memory needs RBAC to prevent cross-user leakage.

How do I prevent memory from growing unbounded?

TTL on memories, LRU eviction, periodic summarization and compaction, per-user storage quotas, and importance scoring that drops low-value memories.

What's the "lost in the middle" problem?

LLMs attend less to information in the middle of long contexts. Relevant memories placed mid-prompt may be ignored. Put critical memories near the start or end of context.

Should memory writes be synchronous or async?

Async for non-critical writes (conversation summaries). Synchronous when the next step depends on the write (checkpoint before tool execution). Never block user response on memory writes.

How do I evaluate memory quality?

Test scenarios: state a preference, continue 20 turns later, verify recall. Cross-session tests. Adversarial tests: verify User A can't retrieve User B's memories.

What's Mem0 and when should I use it?

Mem0 is a memory layer that automatically extracts, deduplicates, and manages agent memories. Use it when you don't want to build write filtering and retrieval logic from scratch.

References

• ReAct: Synergizing Reasoning and Acting (Yao et al., 2022)
• LangChain Agents Documentation
• OpenAI Agents Guide

Further Reading

• Anthropic Tool Use Documentation
• LangGraph Documentation
• Microsoft AutoGen

Summary

• Agent memory spans short-term (messages), working (state), and long-term (vector/episodic) timescales. - Context windows are not memory - externalize state before you hit limits. - Be selective about what you store; unfiltered memory degrades retrieval quality. - Retrieve relevant memories on demand (top_k=3–5), don't accumulate everything in context. - Enforce tenant isolation - memory is user data with privacy and security requirements.

• Use checkpointing for session continuity; vector stores for cross-session recall; summarization for context management. - Memory and RAG share retrieval mechanics but serve different content - combine both in knowledge-heavy agents. - Evaluate memory with cross-turn and cross-session recall tests, not just single-turn accuracy.