Hermes Memory Deep Dive: FTS5, Markdown Files, and Agent-Curated Recall
How Hermes Agent remembers between sessions: markdown files in ~/.hermes, FTS5 full-text search, LLM summarization, and agent-curated persistence.
Most agent memory systems are a black box. Vectors stored somewhere, embeddings you cannot read, recall that works until it does not. Hermes took a deliberately old-school approach: markdown files on disk, indexed by SQLite's FTS5 full-text search, curated by the agent itself. The result is a memory layer that any developer can inspect with cat and grep.
This article is an architecture-level explainer for the Hermes memory system. We will cover where files live, how indexing works, how the agent decides what is worth remembering, and what this design means for trust, privacy, and debuggability.
Key Takeaways
- Hermes memory is a directory of markdown files under
~/.hermes/, not an opaque vector store. - SQLite's FTS5 extension provides full-text search with BM25 ranking and phrase matching.
- The agent itself curates memory: it decides what to persist, what to summarize, and what to discard.
- Memory is human-readable and human-editable — you can open any file in your text editor.
- The design trades some of the recall nuance of vector stores for auditability, portability, and trust.
- This is the architectural complement to the conceptual framing in cross-session memory: fixing the Claude forgets problem.
- If you already wrote your first SKILL.md, this is the next file in the system to understand.
The Layout on Disk
After a few weeks of use, a typical ~/.hermes/ directory looks something like this:
~/.hermes/
├── config.yaml
├── memory/
│ ├── index.db # SQLite FTS5 index
│ ├── projects/
│ │ ├── aiskill-market.md
│ │ └── client-foo.md
│ ├── people/
│ │ ├── jason.md
│ │ └── team-contacts.md
│ ├── procedures/
│ │ └── deploy-checklist.md
│ └── journals/
│ ├── 2026-04-18.md
│ └── 2026-04-19.md
├── skills/
│ └── ...
└── logs/
└── ...
Every file under memory/ is a regular markdown document. Categories like projects, people, procedures, and journals emerge from agent behavior over time, not from a rigid schema. You can open people/jason.md in Vim and read exactly what the agent remembers about me. You can also edit it if something is wrong.
Why Markdown and Not a Vector DB
Three reasons.
Auditability. You can read your agent's memory. When you ask "why did it think the deploy was on a Tuesday?" you can open projects/aiskill-market.md and see that it was, in fact, told the deploy was on a Tuesday. No embedding space to spelunk through.
Portability. A ~/.hermes/ directory is a git repo waiting to happen. You can back it up with rsync, commit it to a private repo, copy it between machines, and diff it against last month's state.
Trust. When the memory layer is text, you retain consent. You can delete a line, redact a section, audit what the agent knows about third parties. A vector store gives you none of that without running queries you may not know to ask.
The tradeoff is that markdown plus FTS5 cannot match the semantic nuance of a well-tuned embedding store. Hermes leans into that tradeoff deliberately.
How FTS5 Fits In
SQLite's FTS5 is a full-text search extension that supports phrase matching, prefix queries, proximity operators, and BM25 ranking out of the box. Hermes maintains an index.db alongside the markdown files. When the agent writes or edits a memory file, it triggers a re-index of that file. Searches like hermes memory search "deploy checklist" go through FTS5 and return a ranked list of files with highlighted snippets.
A simplified version of the query shape Hermes uses internally:
SELECT path, rank, snippet(memory_index, 1, '<mark>', '</mark>', '...', 20)
FROM memory_index
WHERE memory_index MATCH 'deploy NEAR checklist'
ORDER BY rank;
FTS5 is fast enough that an index over hundreds of files returns results in single-digit milliseconds on a $5 VPS.
Agent-Curated Persistence
This is the part that separates Hermes memory from most competing designs. Memory is not automatic. The agent decides what to persist.
During a chat session, Hermes accumulates short-term context normally. At natural break points — end of a skill, explicit save instructions, session close — the agent invokes its memory.write tool with three choices to make: what file to write into, what to add, and what to summarize or collapse from existing content. A naive implementation would append everything. Hermes is opinionated about compression: new facts merge into existing sections, outdated claims get edited in place, and entire files get rewritten when their structure drifts.
This is why the memory directory stays readable over time. It is not a transcript log; it is a curated encyclopedia the agent maintains about its world.
How Memory Gets Used in a Session
At session start Hermes runs a short retrieval step. Given the user's first message, it queries FTS5 for potentially relevant files, loads the top matches into context, and then enters the regular chat loop. Mid-session, the agent can invoke memory.search explicitly to pull more context as needed. This is different from "always load everything" and different from "only load via explicit @mentions." It is closer to how a human looks things up as needed, powered by FTS5 ranking.
The relevant tool calls you will see in logs:
memory.search query="deploy checklist"
memory.read path="procedures/deploy-checklist.md"
memory.write path="projects/aiskill-market.md" merge="true"
Implications for Trust and Privacy
A few practical consequences of this design.
Secrets do not belong in memory. Because files are human-readable, the same ease-of-audit means anything written there is exposed to any process with read access to ~/.hermes/. Hermes explicitly warns agents against writing API keys or tokens into memory files. Keep those in config.yaml or a secret manager.
Team use requires conscious sharing. A single Hermes instance's memory is tied to that machine. Sharing across a team means sharing the memory/ directory, typically via a private git repo. That is a feature if you want your team to have a common agent brain; a liability if you did not realize you were signing up for shared context.
Deletion is real. rm ~/.hermes/memory/projects/client-foo.md actually removes the file. FTS5 re-indexes on next write. There is no shadow embedding carrying the content forward. For compliance-sensitive workflows, this is a meaningful property.
Comparing to Other Memory Designs
For the full architectural comparison against static CLAUDE.md, vector databases, and raw chat transcripts, see cross-session memory: how Hermes fixes the Claude forgets problem. The short version: each design optimizes for a different property. Static files are predictable but stale; vector DBs are flexible but opaque; chat transcripts are complete but noisy. Hermes's markdown-plus-FTS5 approach lands on auditable, editable, and searchable, with curation as the active ingredient.
Operating the Memory Layer
Day-to-day commands you will actually run:
# List what the agent knows about
hermes memory list
# Search by keyword
hermes memory search "weekly review"
# Open a specific memory file in your editor
hermes memory edit projects/aiskill-market.md
# Back up the whole memory directory
tar czf hermes-memory-backup-$(date +%F).tgz ~/.hermes/memory/
The fact that hermes memory edit just launches $EDITOR on a markdown file tells you something about the philosophy.
What Is Next
The architectural piece you just read explains the how. The conceptual piece, cross-session memory: fixing the Claude forgets problem, explains the why and where this fits against other memory strategies. If you have not yet authored a SKILL.md, loop back to writing your first Hermes SKILL.md — skills and memory work hand in hand.