I’ve tried a lot of note-taking and knowledge-management systems over the years — different apps, different tools, different organizing principles, each one adopted with real enthusiasm. None of them stuck. Looking back, I think the reason is the same every time: the friction of capturing a thought, then organizing it, then periodically re-organizing everything as my sense of the “right” structure changed, was just enough to make the initial burst of enthusiasm fade. The system would keep working; I’d just quietly stop feeding it, and a few months later I’d have a knowledge base gathering dust instead of a second brain.
But the underlying need never went away — I still need a place to put notes, and I need a way to get them back that doesn’t cost more effort than it saves. So I built another system. This one is new enough that I can’t yet claim it’s the one that finally sticks, but it’s showing real promise. The reason is that it entirely removes the decision of “Where should I put this (within my knowledge base)?” entirely rather than making it easier. I send a thought — typed or dictated, from whatever chat app I have open — and it gets filed in the right place, with the right tags etc.. I ask a question later, in plain language, and it comes back. Anything with a deadline turns into a reminder that reaches every device I own. I never choose a folder location, a tag, or a format.
I call it my working memory system, it runs on top of Hermes — a personal agent I host on my own VPS and talk to mostly over Telegram or Hermes’ dedicated apps — and it’s open on GitHub. This post is about how it works.
What it looks like day to day
In a chat I’ve reserved for it, I just talk:
me printer is out of ink
bot ✅ → wiki (record): printer out of ink
me remind me Tuesday at 8am to call the plumber
bot ✅ → Todoist: call the plumber (Tue 08:00)
me BP 128/82 this morning
bot ✅ → wiki (series): blood-pressure
me what's my BP been doing?
bot Six readings since the 12th, trending slightly up — 128/82 this
morning versus 121/79 two weeks ago. The two highest were both
after evening coffee.
That last exchange is quite interesting. The blood-pressure readings live in one small markdown file, one line per reading. The Hermes agent just reads the file and reasons over it, the way a person would if they kept a paper log and reread it before a doctor’s visit. There’s no query language that expresses “the two highest were after evening coffee,” but a model reading forty lines of text notices it without being asked to.
Anywhere else — any chat Hermes is connected to, not just the reserved one — I just prefix with Hey memory:
me Hey memory, the plumber's number is 555-0134
bot ✅ → wiki (reference/entity): plumber
I dictate a fair number of these with VoiceInk on my Mac or the OS-level dictation on my iPhone. With voice dictation, capturing the information is as easy as it gets. That’s crucial, because if it takes much longer to store a thought than to have it, I won’t store it.
Retrieval based organization
The one genuinely opinionated idea in this system is that notes shouldn’t be organized by subject at all. Rather, they should be organized by how the information in them needs to be retrieved. This is described in detail in my blog post about Retrieval-First Note Organization). Briefly, any information that gets stored is classified into one of five distinct types: Reminder, Record, Reference, Project, or Idea with a Reference type having subtypes as Entity, Concept, Procedure. And that’s it. Everything I have noted for the past decade fits neatly into this schema.
How it works
The Hermes agent determines what type the message I send it is. Reminders get stored as tasks in Todoist, via its API. The other types get stored as notes in my Obsidian vault. Each note gets a tag from an existing canonical list of tags; if there’s not good tag in the canonical list, the agent creates a new tag and updates the canonical list with it. Once a week, the agent runs a “linter” script on the wiki that checks if content or tags need reorganization, creates links, updates indexes etc.
A capture gate decides whether a message is memory input at all — because it’s in a chat I reserved, or because it starts with Hey memory. Everything else falls through as ordinary conversation, untouched. Matching messages buffer for a few seconds so that three thoughts typed in a row become one turn instead of three separate ones.
The agent does the actual classification and routing, following a policy document that lives in the repo as plain English instructions rather than code. This is deliberate — the parts of this system that require judgment (is this a project or an idea? does a reference page need to flip from active to superseded because something newer replaces it? does this sentence even want to be remembered?) are exactly the parts I don’t want to hand-code, because I’d be re-deriving what a language model already does well.
A small set of deterministic tools handles everything that shouldn’t require judgment — talking to Todoist, file locking, nightly backups. The agent calls these rather than improvising each step, so a transcript entry comes out the same shape every time regardless of what triggered it.
The notes in the obsidian vault are pushed to a private git repo after every write. The agent also does a git pull before it accesses something from the vault. The Obsidian app is available on my Mac and iPhone and it syncs data to/from the git repo via native git plugin (on MacOS) and via Working Copy (on iPhone and iPad). This way, my entire knowledge base is available on all my devices. Querying happens by direct manual reads or via Hermes (e.g. “Hey memory, how did I embed subtitles in a .mkv file and set the default subtitle track?”). Writing happens mostly by telling Hermes, and sometimes manually.
Conclusion
These days I just talk to my Mac on iPhone when asking it to store things to memory and when querying what I had stored in the past. It really is frictionless. The lack of friction has been instrumental in me engaging more with my knowledge base, and letting it compound richly. The friction was eliminated by creating a retrieval based schema for storing the knowledge and automating the tedious bits of classification, tagging, storage, and retrieval with Hermes agent, who’s always available on all my devices.
Working with my knowledge base throughout the day now feels natural, rather than a forced ritual.