Session Memory vs Long-Term Memory: Picking the Right Layer

The question teams ask when they hear about agent memory is almost always "which framework should we use." The question they should ask first is "do we need one at all." A surprising number of AI products bolt on a persistent memory layer — with all its retrieval logic, storage costs, and privacy surface area — for a use case that a well-managed conversation window would have handled just fine.

Memory, in the agent-systems sense, comes in at least two distinct layers, and conflating them is where most architecture decisions go wrong. Session memory is what an agent needs within a single conversation or task: the last several exchanges, the current task state, whatever context makes the next response coherent given what was just said. Long-term memory is what an agent needs to persist across sessions entirely: facts about the user, history from prior interactions, anything that should still be true and available next week, next month, or next year.

Treating these as one problem — "memory" — instead of two, with different failure modes and different solutions, is the single most common source of over-engineering in this space.

Session memory is mostly a context-window problem

For the duration of a single conversation, an agent mostly needs what's already in its context window, managed sensibly. This isn't really a "memory system" problem in the durable-storage sense — it's a context management problem: how much of the conversation history to keep verbatim, when to summarize older turns to save tokens, how to preserve task state (what step of a multi-step process the agent is on) without re-deriving it from scratch every turn.

Most agent frameworks handle this adequately out of the box, because it maps directly onto how LLM APIs already work — you're managing what goes into the next prompt, not building a database. The mistake to avoid here is reaching for a full persistent memory framework to solve what is, in reality, a sliding-window and summarization problem. If your agent's job is done the moment the conversation ends — a one-off coding task, a single customer support ticket resolved in one sitting — durable long-term memory is solving a problem you don't have.

Long-term memory is a different problem with a different cost

The moment a product needs to recognize a returning user, recall a preference from a prior session, or reason about how a situation has changed since last time, session memory alone can't do it — that context is gone the instant the conversation ends unless something persisted it. This is where the dedicated frameworks earn their place: Mem0 as a memory layer purpose-built to give agents personalized memory that persists across sessions and evolves over time; Zep and its Graphiti engine for cases where facts change and the history of that change matters; Cognee when relationships between facts need to be queryable, not just individually retrievable.

Long-term memory comes with costs session memory doesn't: storage that grows indefinitely, retrieval logic that has to find the needle in an accumulating haystack, decisions about staleness and forgetting (a problem serious enough for its own dedicated piece), and privacy obligations that only apply once you're persisting personal information beyond a single interactive session.

A practical decision framework

The question worth asking before reaching for any memory framework is simple: does this agent's usefulness depend on knowing something about a specific user or situation that outlives a single session? If the answer is genuinely no — a stateless utility that does the same well-defined thing for anyone who calls it, a tool that operates entirely on inputs provided fresh each time — session memory, managed well, is the entire solution. Building durable storage on top of that is pure overhead: cost without a corresponding benefit, and a privacy liability for data you never needed to keep.

If the answer is yes, the next question is how much needs to persist and how structured does it need to be. A simple product — a journaling app that recalls a user's past entries — might get by with a plain vector store indexed by user ID, no graph, no temporal reasoning required. A more relational product — a project-management assistant that has to track how a plan changed over multiple stakeholder conversations — starts to need the structure that graph-native and temporal memory layers provide, for the reasons covered in Why Graph-Native Memory Is Winning Over Pure Vector Stores.

The hybrid case is the common case

In practice, almost every product that needs long-term memory also needs session memory — they're not mutually exclusive, they're layered. A customer support agent needs to track the current ticket's back-and-forth (session memory) while also recalling that this customer had a billing dispute three months ago (long-term memory). Building only the long-term layer and neglecting session-level context management produces an agent that "remembers" the user's history but loses track of what was said two messages ago in the current exchange — an oddly specific and confusing failure mode that users read as the agent being distracted rather than forgetful.

Letta's core-memory-and-archival-memory split is instructive here even outside its specific implementation, because it effectively encodes both layers into one runtime: core memory functions like a durable, always-present session anchor, while archival memory is the long-term store proper. Thinking in terms of that split — what needs to be present right now, versus what needs to be retrievable eventually — is a useful mental model whether or not you adopt Letta specifically.

Don't build the long-term layer before you know what to store

One quiet trap in this decision is committing to a long-term memory architecture — graph database, temporal store, vector index with a particular embedding model — before you actually know what kinds of facts your product needs to remember. Early-stage products often don't know yet whether their memory needs are relational (facts connect to each other), temporal (facts change over time), or simple (facts are independent and static). Building the sophisticated version prematurely means paying its operational complexity before you've validated it's the complexity you actually need.

A reasonable sequencing: start with session memory done well, add a minimal long-term layer (even something as blunt as a Postgres table of user facts with basic retrieval) once you have real evidence users benefit from persistence, and only reach for graph-native or temporal architectures once you can point to specific interactions that plain storage handled badly. This mirrors the broader build-versus-buy calculus covered in Building Your Own Memory Layer vs Buying One — the cheapest mistake to avoid is architecting for a scale and sophistication of memory your product hasn't earned yet.

The layer choice is a product decision, not just a technical one

Ultimately, the session-versus-long-term question isn't purely an engineering call — it reflects a real product decision about what kind of relationship your agent has with its users. A tool is used and forgotten. A companion is used and remembered. Neither is wrong, but building the memory architecture for the wrong one — durable memory for a tool, ephemeral context for a companion — produces a product that either wastes engineering effort on persistence nobody benefits from, or frustrates users by forgetting things they reasonably expected it to remember. Get clear on which relationship you're building before you get clear on which framework to use.

Part of the "Agent Memory" series on aiskill.market.