A Memory Architecture for a Solo Builder's First Agent

If you've read through the rest of this series, you now know that agent memory can involve temporal graphs, tiered core-and-archival storage, contradiction resolution, and a genuine competitive-moat calculus. None of that is wrong, and all of it is real engineering work happening at real companies right now. It is also, almost certainly, not what you should build if you're one person shipping your first agent this month. The gap between "what's technically the right architecture" and "what a solo builder should actually build first" is the whole subject of this piece.

The honest starting point: most first agents fail to find product-market fit before they fail because of inadequate memory infrastructure. Over-investing in memory sophistication before you know whether anyone wants your product is a well-disguised form of procrastination — it feels like serious engineering, and it's often just avoidance of the harder, less technical work of figuring out if the thing is useful at all. So the goal here isn't the best possible memory architecture. It's the smallest one that won't embarrass you in front of your first real users.

Start with nothing, on purpose

Before building any persistence at all, ask honestly whether your agent needs memory beyond a single conversation. A surprising number of genuinely useful first agents don't — a tool that helps draft an email, summarizes a document, answers a coding question. If the value delivered in one sitting is complete in itself, session memory (just managing the current conversation's context window sensibly) is the entire memory architecture you need, and everything covered in Session Memory vs Long-Term Memory about durable storage doesn't apply to you yet. Building persistence you don't need is the single most common over-engineering trap for a first agent, precisely because persistent memory is the part of this space that gets talked about the most.

The minimal stack that actually works

If your agent genuinely does need to remember things across sessions — recognizing a returning user, recalling a stated preference, picking up where a prior conversation left off — here's a stack that a solo builder can stand up in a few days and that will not need to be thrown away when real usage arrives.

A single Postgres table, keyed by user ID, storing extracted facts as rows: the fact itself, when it was learned, and a simple category tag (preference, biographical, project-context, whatever taxonomy fits your product). Add pgvector and embed each fact so you can do similarity search when a conversation needs to pull in relevant history. That's it. No graph database, no dedicated memory service, no framework integration. This gets you real persistent memory — the actual UX payoff of "it remembers me" discussed in Memory Is the New UX Pattern for AI Products — with infrastructure you already know how to operate.

For extraction — deciding what's worth remembering from a conversation — a single LLM call at the end of a session, asked to identify durable facts worth persisting, is enough to start. It won't be perfect. It doesn't need to be; it needs to be good enough that your early users notice the agent remembering things that matter to them, not zero-defect fact extraction from day one.

Build the tiny version of the two-tier split

You don't need Letta's runtime to benefit from its core insight. Manually maintain a short, hand-curated list of facts that always go into every prompt — the user's name, one or two standing preferences you've decided matter enough to never look up. Everything else lives in the Postgres table, retrieved via similarity search only when the current conversation seems related. This gives you the always-in-prompt-versus-queryable-on-demand split covered in Core Memory vs Archival Memory without adopting any specific framework — just a deliberate choice about which handful of facts earn permanent placement.

Keep this list genuinely short. The temptation, as you learn more about a user, is to keep adding things to the "always inject" set, and that's exactly the bloat that tiering exists to prevent. If it's not something that matters in nearly every conversation, it belongs in the queryable store, not the permanent one.

Decide your forgetting policy before you need one

You don't need sophisticated decay scoring for a first agent, but you do need an answer, decided in advance, to two questions: what happens when a new fact contradicts an old one, and does a user have a way to tell the agent to forget something. For the first, the simplest workable rule is recency-wins — a new fact on the same topic supersedes an old one, and you keep both rows with timestamps rather than overwriting, so you at least have the history if you need it later. For the second, even a blunt "delete all facts for this user ID" command, invokable on request, is enough to start — it won't satisfy every nuance covered in The Privacy Question Nobody Answers When They Ship Agent Memory, but having something deliberate beats having no answer when a user asks.

What to explicitly not build yet

Skip a graph database. Skip temporal-graph reasoning. Skip a dedicated memory framework integration. Skip relevance-decay scoring beyond "new facts win over old ones on the same topic." All of these are real, valuable pieces of engineering — for a product with real usage history and a specific, evidenced need for the structure they provide. For a first agent, they're solving problems you don't have data to know you have yet, and every hour spent on them is an hour not spent finding out whether anyone wants to use the thing at all.

The single clearest signal that you've outgrown this minimal stack isn't a feeling that it's "not sophisticated enough" — it's a specific, concrete failure a user actually hit: the agent contradicted itself in a way that annoyed someone, or a feature you want to build genuinely requires reasoning about how facts relate to each other in a way flat storage can't support. Wait for that signal. It's a far better guide than trying to anticipate every architectural need before you've shipped anything at all.

When to graduate, and to what

If usage grows and you start hitting the specific failure modes this minimal stack can't handle — facts that need temporal reasoning because your product genuinely tracks change over time, relationships between facts that a flat table can't express, retrieval quality visibly degrading as memory volume grows per user — that's your signal to look at the heavier options: Mem0 if you mainly need a more capable, purpose-built persistence layer without wanting to own the extraction and retrieval logic yourself; Zep and Graphiti specifically if temporal reasoning about changing facts is now core to your product; Cognee if relational, multi-hop reasoning between facts is what your users actually need; Letta if you want the core-and-archival tiering formalized into a real runtime rather than the hand-rolled version described above. The build-versus-buy tradeoffs at that point are covered in more depth in Building Your Own Memory Layer vs Buying One — but you'll be making that decision with real evidence about your product's actual memory needs, which is a fundamentally better position than guessing from a standing start.

The point of starting small

A minimal memory stack isn't a compromise you'll regret — it's the correct architecture for a product whose actual memory requirements you don't know yet, because nobody's used it enough for those requirements to reveal themselves. Ship the Postgres-and-embeddings version. Let real users teach you what your agent actually needs to remember, and how that memory needs to behave, before you build infrastructure sized for problems you're currently only imagining. The sophistication described everywhere else in this series is worth having eventually. It's not worth having on day one, and the discipline to know the difference is the actual skill.

Part of the "Agent Memory" series on aiskill.market.