From Prompts to Persistence: Agent Memory, Context, and Memory Engineering

Most agent demos are amnesiac by design. You send a message, the agent responds, and the next time you open a session it has no idea who you are. That is fine for a toy. It is a fundamental problem for anything you want to run in production, where the whole point of an agent is that it accumulates context over time and uses it.

Memory is where the gap between a compelling demo and a reliable system actually lives. Not the model, not the tools, not the orchestration. The memory.

What we mean when we say memory

The word gets overloaded fast. In the context of AI agents, memory roughly covers three different things that behave very differently.

Working memory is the context window. Everything the model can see right now. It is fast, it is immediate, and it vanishes the moment the session ends. Most agent frameworks treat this as "memory" and leave it at that. That works until your conversation runs long, your context fills up, and you start truncating the oldest messages to fit. At that point you are making an implicit decision about what the agent should forget, and you are making it badly.

Episodic memory is a record of what happened. Past conversations, actions taken, decisions made. The equivalent of a log. You can retrieve episodes by similarity, by recency, or by some combination. The agent uses them to reconstruct context across sessions: "Last time this user asked about deployment pipelines, here is what we discussed."

Semantic memory is structured knowledge: facts, preferences, entities the agent has learned about a user or domain over time. "This user works at a fintech company, prefers TypeScript, and has asked about Kafka twice." That is not an episode, it is a derived fact you want to keep updated as new information comes in.

The mistake most engineers make is building one store and cramming everything into it. Episodic and semantic memory have different retrieval patterns, different update frequencies, and different lifetimes. Mixing them makes both worse.

Where you actually store this

The storage layer is a solved problem with well-understood trade-offs. The question is matching the store to the access pattern.

For episodic memory, you almost always want a vector database or a hybrid store that combines vector similarity with metadata filtering. The retrieval question is "what past events are relevant to what is happening right now?" and that is a semantic similarity problem. But raw vector search without metadata filters is noisy. Adding filters for user ID, recency, topic, or session type dramatically sharpens retrieval. The combination of a dense embedding plus a sparse metadata filter is what makes retrieval actually useful rather than just adjacent.

For semantic memory, a document store works well. Facts about a user or entity are updated in place, not appended to. The schema is flexible because what you know about a user in week one is different from what you know in month six. You want to read and write individual facts efficiently, not scan a vector index for them.

Working memory management is a code problem, not a storage problem. You need a strategy for what to include in the context window and in what order. A common pattern: the most recent N turns verbatim, plus a retrieved summary of older relevant episodes, plus key semantic facts about the user. That structure fits more useful context into a fixed token budget than naive full-history approaches.

Retrieval is the hard part

Storing memories is straightforward. Knowing when to retrieve them, and which ones, is where most implementations go wrong.

The obvious approach is "retrieve memories that are similar to the current user message." That sounds right and works poorly in practice. The current message might be short, ambiguous, or phrased in a way that does not match how the memory was originally stored. You end up retrieving things that match the words but miss the intent.

Better retrieval mixes several signals. Semantic similarity is one. Recency matters too: a memory from yesterday is usually more relevant than one from three months ago, even if the older one scores higher on embedding similarity. You can combine these with a weighted rank rather than relying on any single score.

There is also a specificity problem. Agents often have both generic knowledge and user-specific memories. When the two conflict or overlap, you need to decide which to surface. A memory that is highly specific to this user and this context should outrank a vague general episode, even if the general one is semantically closer to the query.

The teams that get retrieval right tend to iterate on it empirically. They log what was retrieved, what the agent used, and what would have been more useful. That feedback loop, built deliberately, is what separates a retrieval system that gets better over time from one that stays mediocre indefinitely.

Memory lifecycle: the part nobody thinks about until it breaks

Creating memories is the easy part. The hard part is deciding when to update them and when to delete them.

User preferences change. A fact that was accurate six months ago might be wrong now. If your agent confidently recalls that a user prefers framework X when they have since switched to framework Y, it is not helpful, it is annoying. Staleness is a bug, and it gets worse the longer the agent runs.

The update problem has a few patterns worth knowing. For semantic facts, update-in-place is usually right: you do not want fifty versions of "user's preferred deployment platform," you want the current one. For episodic memory, append-only is more appropriate: you want to keep the record of what happened, not overwrite it.

Pruning matters more than most teams expect. An agent that accumulates memories indefinitely will either degrade in retrieval quality (too much noise) or run up storage and latency costs. You need a policy for what to keep. Recency is the simplest: keep the last N episodes. Importance scoring is more sophisticated: use the model to score how significant a memory is and prune low-scoring old ones. Neither is perfect, and the right choice depends on your use case.

One thing that does not work: ignoring pruning until retrieval latency starts climbing and then doing a one-off cleanup. By then you are reacting instead of designing.

The practical architecture

If you were building this from scratch, a reasonable starting point looks like this. A vector store (MongoDB Atlas, Pinecone, pgvector) for episodic retrieval. A document database for semantic facts. A working memory builder that assembles context by combining the most recent turns, retrieved episodes, and relevant semantic facts into a prompt prefix before each LLM call.

The orchestration logic decides: after each turn, should we store anything? If the user stated a preference or the agent learned something new, write a semantic memory. If the turn contained meaningful content worth recalling later, write an episode. Keep that logic simple and explicit. An agent that writes a memory for every turn is creating noise. An agent that only writes memories when you explicitly tell it to is brittle.

What you are building is not that different from a caching layer, an event log, and a user profile store. Those are infrastructure patterns engineers have been building for years. The fact that the consumer is an LLM instead of a service does not change the fundamental design questions.

The teams I have seen do this well are the ones who stopped treating memory as an AI feature and started treating it as a data engineering problem. Schemas, access patterns, retention policies, retrieval quality metrics. The same discipline that makes a database fast and reliable makes an agent memory system trustworthy.

If you are building agentic systems and want to think through the architecture, I run one-on-one mentorship sessions for senior engineers working through exactly these problems. And the Monday BY Gazar newsletter covers system design decisions like this one every week.