An agent's memory is everything it can use beyond the current message: the conversation so far, notes it saved, and facts it can look up again. The working rule is to keep the context window small and high-signal, move anything that must outlast a session into storage outside it, and pull it back only when a task needs it.
Most memory bugs come from getting one of those three wrong. The agent keeps too much and gets worse as the context grows, or it summarizes too hard and loses the one detail that mattered, or it saves everything and recalls the wrong thing. This post goes through each part with working code.
Short-term and long-term memory
It helps to split memory in two.
Short-term memory is the context window: system prompt, tool definitions, conversation history and tool results. The model sees all of it on every turn, and it disappears when the session ends unless you store it.
Long-term memory lives outside the model. It can be a file of notes, a database table or a vector index. The model never sees it directly. It has to be written on purpose and loaded back on purpose.
Agent frameworks blur this a little. The OpenAI Agents SDK, for example, has sessions that store the conversation in SQLite or Redis and replay it before every run. That is persistent, but it is still short-term memory in the sense that matters: it all goes back into the context window.
Why can't the agent just keep everything?
Two reasons: cost and quality.
Cost is simple. Every token in the history is billed again on every turn. If you have not set up prompt caching, a long history gets expensive fast.
Quality is the bigger problem. Anthropic's applied AI team calls it context rot: as the number of tokens in the window grows, the model's ability to recall information from that context goes down. They describe context as a finite resource with diminishing returns, and say the decline is a gradient, not a hard cliff.
Position matters too. In "Lost in the Middle", Liu and colleagues found that models often do best when the relevant information is at the start or end of the input, and noticeably worse when it sits in the middle of a long context. A fact buried 40 turns back is in exactly that spot.
So the goal is not the biggest context you can afford. Anthropic puts it as the smallest set of high-signal tokens that gets the job done.
Trim, compact or clear?
There are three common ways to shrink a history. They lose different things.
- Trimming drops old turns and keeps the last N. It is cheap and predictable, and it forgets everything older than the cutoff.
- Compaction replaces the history with a summary and continues from there. It keeps more, but whatever the summary leaves out is gone.
- Clearing tool results removes the raw output of old tool calls and keeps the rest. Anthropic calls this the safest, lightest-touch form of compaction, because a tool result the agent already acted on rarely needs to be read again.
Clearing is a good first step, since tool output is usually the bulk of an agent's context. Compaction comes next, when even the conversation itself is too long.
Trimming with the OpenAI Agents SDK
A session's history can be filtered per run with session_input_callback. This is the example from the SDK's documentation. The callback gets the stored history and the new turn, and whatever it returns is what the model sees:
from agents import Agent, RunConfig, Runner, SQLiteSession
def keep_recent_history(history, new_input):
# Keep only the last 10 history items, then append the new turn.
return history[-10:] + new_input
agent = Agent(name="Assistant")
session = SQLiteSession("conversation_123")
result = await Runner.run(
agent,
"Continue from the latest updates only.",
session=session,
run_config=RunConfig(session_input_callback=keep_recent_history),
)
The stored session is not rewritten. The SDK only saves the items from the new turn, so you can change the filter later without losing anything. If you only need a cap, RunConfig(session_settings=SessionSettings(limit=50)) does the same thing for the most recent 50 items.
For summaries, the SDK has OpenAIResponsesCompactionSession, which wraps another session and compacts the stored history with the Responses API after each turn once a threshold is met. Unlike the callback, compaction does clear and rewrite the stored history.
Clearing tool results on Claude
On the Claude API, context editing does the clearing on the server. It is in beta and needs the context-management-2025-06-27 header:
response = client.beta.messages.create(
model=MODEL, # any current Claude model
max_tokens=4096,
messages=messages,
tools=tools,
betas=["context-management-2025-06-27"],
context_management={"edits": [{"type": "clear_tool_uses_20250919"}]},
)
By default it starts clearing once the input passes 100,000 tokens and keeps the three most recent tool uses. You can change both, and exclude_tools lets you protect results the agent will need again.
What a good summary keeps
Anthropic describes how Claude Code compacts: it keeps architectural decisions, unresolved bugs and implementation details, drops redundant tool output, and carries on with the summary plus the five files it accessed most recently.
That is a useful template for any agent. A summary should keep decisions and the reasons for them, open problems, exact identifiers (IDs, file paths, order numbers, names), and anything the user asked for that is not done yet. It can drop tool output the agent already acted on, dead ends and small talk.
Anthropic's advice for tuning a compaction prompt is to aim for recall first, so nothing important is lost, and then trim for precision. Summaries that are too aggressive lose details that only turn out to matter later.
Notes outside the context window
Long-term memory starts with the agent writing things down. Anthropic calls this structured note-taking: the agent keeps a to-do list or a NOTES.md file outside the context and reads it back later. Their example is Claude playing Pokemon, keeping tallies, maps and strategy notes that survive context resets without being told how to structure them.
On the Claude API, the memory tool gives this a standard shape. You add {"type": "memory_20250818", "name": "memory"} to your tools, and Claude asks to view, create, edit, rename and delete files under a /memories directory. Your code does the actual file work, so the data stays in your own storage. The API also tells Claude to check that directory before it starts a task.
Because the model chooses the paths, the handler has to stop it from leaving the directory. The documentation gives /memories/../../secrets.env as the attack to block, and recommends resolving paths and checking they stay inside the root. Here is a handler that does that:
from pathlib import Path
from urllib.parse import unquote
MEMORY_ROOT = Path("./memory_store").resolve()
def safe_path(requested: str) -> Path:
"""Map a /memories/... path from the model onto disk, or raise."""
decoded = unquote(requested)
if decoded != "/memories" and not decoded.startswith("/memories/"):
raise ValueError(f"path must start with /memories: {requested}")
relative = decoded[len("/memories"):].lstrip("/\\")
target = (MEMORY_ROOT / relative).resolve()
target.relative_to(MEMORY_ROOT) # raises ValueError if it escaped
return target
It refuses ../ tricks, URL-encoded ones like %2e%2e, and paths outside /memories. The docs also suggest capping file sizes and deleting files that have not been read in a long time.
Pairing the two features works well. With context editing on, Claude gets a warning as the context nears the clearing threshold, so it can save what matters to memory before old tool results are removed.
How should an agent decide what to recall?
Once there are more notes than fit in the context, the agent has to pick. The best-known scheme comes from the Generative Agents paper by Park and colleagues. Each memory gets a score that adds three parts, each scaled to the range 0 to 1:
- Recency: decays exponentially since the memory was last retrieved, by a factor of 0.995 per hour of simulated time in the paper.
- Importance: a score from 1 to 10 that the model assigns when the memory is written. In the paper, cleaning up a room is a 2 and asking a crush out on a date is an 8.
- Relevance: similarity between the memory and the current query. The paper uses embeddings.
Here is a small version you can run. It uses word overlap as a stand-in for embeddings to keep it dependency-free, so swap in a real embedding model for production:
import math
import re
import time
from dataclasses import dataclass, field
@dataclass
class Memory:
text: str
importance: int # 1 to 10, rated when the memory is written
last_used: float = field(default_factory=time.time)
def words(s: str) -> set[str]:
return set(re.findall(r"[a-z0-9]+", s.lower()))
def relevance(query: str, text: str) -> float:
# Stand-in for embedding similarity: share of query words found in the memory.
q = words(query)
return len(q & words(text)) / len(q) if q else 0.0
def minmax(xs: list[float]) -> list[float]:
lo, hi = min(xs), max(xs)
return [0.0 if hi == lo else (x - lo) / (hi - lo) for x in xs]
def retrieve(memories: list[Memory], query: str, k: int = 3,
decay: float = 0.995, now: float | None = None) -> list[Memory]:
now = now or time.time()
hours = [(now - m.last_used) / 3600 for m in memories]
rec = minmax([decay ** h for h in hours])
imp = minmax([float(m.importance) for m in memories])
rel = minmax([relevance(query, m.text) for m in memories])
scored = sorted(zip(memories, (r + i + v for r, i, v in zip(rec, imp, rel))),
key=lambda pair: pair[1], reverse=True)
top = [m for m, _ in scored[:k]]
for m in top:
m.last_used = now # retrieval refreshes recency
return top
Asked "when should I deploy the billing fix?", it returns the staging-cluster memory, the 2 MB billing limit and the deploy window, and skips the recent but unimportant "thanks". Retrieval also refreshes a memory's recency, so facts that keep being useful stay easy to find.
What should an agent forget?
Forgetting is a policy, not an accident. A few rules cover most cases:
- Expire what is only true for now. A branch name, a temporary error or today's status should carry a date and be dropped when it stops mattering.
- Replace, do not append. When a fact changes, update the old note. Two contradictory notes are worse than none, because the agent may pick the wrong one.
- Let low scores fade. With the scoring above, a low-importance memory that is never retrieved sinks on its own. Delete it after a set time instead of keeping it forever.
- Never store secrets. Passwords, keys and card numbers should not go into memory at all. The memory tool docs say to strip sensitive data before writing.
- Forget on request. If a user asks the agent to forget something, delete it, including anything derived from it.
Memory also feeds tool use. A memory store is one more tool, and a confusing one causes the same problems as any badly described tool. The post on why agents call the wrong tool covers how to name and describe tools so the model uses them correctly, and the evals guide is a good way to check that a memory change actually helped.
Checklist
- Clear old tool results before you summarize anything.
- Write compaction prompts for recall first: decisions, open problems, exact IDs.
- Save facts that must outlast a session to storage outside the context.
- Check every memory path stays inside its root directory.
- Score recall on recency, importance and relevance, not recency alone.
- Date short-lived facts, update changed ones, and delete what is stale.
- Keep secrets out of memory entirely.
FAQ
What is the difference between short-term and long-term memory in an AI agent?
Short-term memory is what is in the context window on the current turn. Long-term memory is stored outside the model, in files or a database, and has to be loaded back into the context before the model can use it.
Does the Claude memory tool store data on Anthropic's servers?
No. The memory tool runs on the client side. Claude asks for file operations and your application carries them out, so you choose where the data is stored and how long it is kept.
When should an agent summarize its conversation?
When the history gets long enough to hurt cost or quality, and after clearing old tool results is no longer enough. Both the OpenAI Agents SDK and Claude's context editing let you set the threshold.
Is trimming or summarizing better?
Trimming is predictable and cheap but forgets everything older than the cutoff. Summarizing keeps more but can drop details. Many agents trim or clear tool output first and summarize only when that is not enough.
Sources
- Anthropic, Effective context engineering for AI agents (engineering blog, September 2025)
- Anthropic, Memory tool (Claude API documentation)
- Anthropic, Context editing (Claude API documentation)
- OpenAI, Sessions (OpenAI Agents SDK documentation)
- Park et al., Generative Agents: Interactive Simulacra of Human Behavior (2023)
- Liu et al., Lost in the Middle: How Language Models Use Long Contexts (2023)