A large language model has no memory of its own. Every call is stateless; the model “remembers” only the context you include in the current request. Memory design for an Agent is therefore an information-management problem: what deserves to be stored, where it belongs, when it should be retrieved, and when it should be forgotten.
Too many teams choose one of two extremes. Some store nothing, so the Agent starts from scratch every time and users must repeatedly explain the same background. Others store everything, placing every conversation in a vector database until half of the retrieved “memories” are noise. This guide presents a three-layer memory architecture with a clear role for each layer, so teams can implement it incrementally.
The central problem in memory design is not storage but retrieval. An inaccurate memory is worse than no memory at all.
Layer 1: Short-Term Memory—Working State Within a Task
Short-term memory is the content in the current task's context window: the user's goal, conversation history, tool-call records, and intermediate conclusions. This layer already exists naturally. The real design problem is context growth. After a long task runs for dozens of steps, its history may overflow the window or dilute the model's attention. Three techniques are especially useful:
- Rolling summaries: Keep the latest N turns verbatim and compress earlier history into a summary. Preserve critical constraints—such as “Do not touch the config directory” and “The budget limit is 50 yuan”—outside the summary in their original wording. Negative constraints are exactly the details summaries tend to lose.
- A structured state object: Instead of relying on a transcript, explicitly maintain task state: the goal, completed subtasks, outstanding work, verified facts, and unresolved questions. This is more reliable than expecting a model to search its own history, and it is a natural companion to the Plan-Execute pattern.
- Slimmed-down tool results: Extract the relevant points from a full HTML search result or a file containing tens of thousands of lines before adding it to history. Raw tool output is the leading source of context growth.
const taskState = {
goal: "为客户生成 Q2 复盘报告",
constraints: ["只用内部数据", "不发送任何邮件"],
done: ["拉取销售数据", "生成图表"],
todo: ["撰写分析", "人工确认"],
facts: [{ claim: "Q2 营收环比 +12%", source: "bi_query#3" }],
};The lifetime of short-term memory ends with the task. Destroy it when the task is complete. It should not flow automatically into long-term memory; that separation is the first safeguard against contamination.
Layer 2: Long-Term Memory—Facts and Preferences Across Sessions
Long-term memory allows the system to recognize the user next time. It can hold preferences such as replying in Chinese or formatting reports in Markdown, stable facts such as a time zone, role, or project background, and historical decisions such as choosing Option B and the reason for doing so. Its design requires four decisions:
What to store: Keep only entries that are factual, stable, and likely to be reused. Ask whether an item will still be true and useful three months from now. One-off task details do not belong in long-term memory.
How to write it: Use two triggers—an explicit user request such as “Remember that our API prefix is /v2,” or automatic extraction after an Agent task. Automatic extraction must filter aggressively. Permission-related conclusions, such as “The user once said confirmation could be skipped,” must never be stored. That rule is central to preventing memory poisoning. Record the source session with every entry so it remains auditable and deletable.
How to retrieve it: When there are only a few dozen entries, inject all of them into the system prompt. As the collection grows, retrieve only entries relevant to the current task. Each memory should ideally record the last time it was used, which provides a signal for future retirement.
How to forget it: A memory store without an expiration mechanism decays. Outdated preferences and abandoned project conventions continue to mislead the model. Periodically remove entries that have not been used for a long time. When facts conflict, let the newer one supersede the old while retaining an audit trail.
The implementation can be simple: a collection of Markdown files or one database table is enough. Claude Code's CLAUDE.md and the “memory” features in consumer assistants are implementations of this layer at heart.
Quality matters more than volume in long-term memory. A few dozen reliable facts are more useful than a database full of raw conversation logs.
Layer 3: Vector Memory—A Retrieval Layer for Experience at Scale
When the amount of memory is too large to inject in full—thousands of historical support tickets, previous task records, or domain notes—it needs vectorization. Create an Embedding, store it in a vector database, and retrieve semantically relevant entries at the start of a task. Technically, this layer is RAG, so the usual RAG engineering practices apply: chunking, hybrid retrieval, reranking, and threshold-based rejection. See Why RAG Gives the Wrong Answer for a detailed treatment.
Vector memory has two pitfalls of its own:
- Raw conversations perform poorly as memories. They contain noise and unresolved references such as “Do what we just discussed,” which remain useless when retrieved later. Distill before storing: compress a completed task into a structured “scenario + decision + result” entry. The improvement in retrieval quality is dramatic.
- Conflicts over time: Semantic retrieval does not understand that a new policy supersedes an old one, so both may be recalled. Add a timestamp and topic key when writing each entry. Keep only the latest version for a topic, or apply recency weighting when injecting results.
An advanced option is graph memory, where entities and relationships are stored in a knowledge graph—Person A owns Project B, and Project B depends on Service C. It fits domains with dense relationships. Open-source projects such as Zep and Mem0's graph mode are moving in this direction, but most applications should get the first two layers right before adding it.
How the Three Layers Work Together: Memory Flow Through One Task
Choose the Depth of Implementation by Product Type
| Product Type | Recommended Configuration |
|---|---|
| Single-turn tool assistant | Short-term memory only, perhaps just the conversation history |
| Personal assistant / Copilot | Short-term + long-term; a file or table is enough for preferences |
| Customer-service / domain-expert Agent | All three layers, with distilled ticket experience in the vector layer |
| Multi-Agent system | All three layers + read/write permissions for shared memory |
Existing options include Mem0, Zep, and LangGraph's checkpointer and store. Before adopting one, focus on three questions: Can you control the write rules? Can you audit and delete an individual memory? Does retrieval support recency weighting? A memory framework solves the engineering problem of storage and retrieval. You still have to decide what to remember and what to forget.
Define the write and retirement rules for each layer before choosing a framework. Reverse that order and you will accumulate memories the system cannot use.
Who This Is For and an Alternative Worth Considering
This guide is for developers adding memory to an Agent or assistant product. One alternative deserves serious consideration: many requirements described as “memory” are better implemented as explicit settings. A preference entered by the user on a settings page is more transparent and controllable than something a model quietly “learns,” and it carries no risk of memory contamination. Letting users view and edit every item the Agent remembers—as if they were maintaining a personal file—currently offers the best balance of usability and safety.
Frequently Asked Questions
Q: Why not put the entire conversation history into a long-context model and avoid designing memory at all? A: It is expensive and slow, and models do not use information in the middle of a long context reliably. See Are Long-Context Models Really Stronger?. A context window is a workbench, not a filing cabinet.
Q: Should memory be isolated by user or shared globally? A: User preferences and personal facts must be isolated per user; that is a privacy baseline. Only de-identified domain experience, such as a general procedure for handling a class of support ticket, should be shared. Writes to the shared layer require stricter review.
Q: How do you evaluate whether memory works well? A: Use two measurable signals. First, does the frequency with which users must repeat background information decline? Second, sample the memories injected into tasks and measure how many are genuinely relevant. If fewer than half are relevant, retrieval is doing more harm than good.
Summary
The three layers have distinct jobs. Short-term memory maintains state and controls context growth. Long-term memory stays selective, auditable, and deletable. Vector memory distills information before storage and uses time to resolve conflicts. The core principle is to manage memory as an asset with a lifecycle: a reliable memory system is one that knows how to forget.