March 7, 2026
AI agents need memory infrastructure
Most AI agent memory is theater. Here is how we built real memory infrastructure: instincts, daily logs, and context engines that survive resets.
The problem

I run an AI operator on top of an agent framework I built. It handles my morning briefings, watches my infrastructure, drafts emails, tracks projects. It runs around the clock with cron jobs that wake it every few hours.
The first thing you learn running an agent in production is that context windows don't persist. Every new session starts blank. Every compaction throws away detail. Every reset means the agent forgets who you are, what you are working on, and whatever it figured out yesterday.
"Memory" in most agent frameworks means stuffing old messages into a vector database and hoping semantic search pulls back something useful. For operational context, it doesn't, at least not reliably. Asking "what did we decide about the backup retention" and getting back three loosely related chat fragments is not memory, just search with extra steps.
What we actually built
After months of the operator forgetting things that mattered, I built a layered memory system. It lives in a plain directory the agent reads on startup, no database, no embeddings.
Daily logs
Markdown files, one per day, memory/2026-03-06.md. Raw observations, decisions, outcomes, written as the session runs. They are ephemeral on purpose. They record what happened, not what matters. There are 72 of them sitting in that directory right now, and most of them I will never read again. The point is that on the one day I need to know what I changed two weeks ago, the file is there.
Promotion to evergreen
When a pattern shows up across enough daily logs, it gets lifted out of the dated files into something durable. In practice the durable layer is two things: a top-level MEMORY.md with the stable facts, and the instinct engine below. The daily logs are the raw feed. The promotion step is where the agent decides what is worth carrying forward, so the signal survives and the noise ages out with the dated files.
Instincts
This is the layer I am proudest of. Instincts are confidence-weighted behavioral rules stored as YAML, one file per learning. Here is the actual format the engine reads, from the instinct system's own README:
id: <short-kebab-case-id>
created: YYYY-MM-DD
updated: YYYY-MM-DD
confidence: 0.5 # 0.3 (weak) to 0.9 (proven)
domain: <config|coding|security|orchestration|communication|infra>
trigger: "<when does this apply>"
evidence_count: 1
action: |
<concrete action to take>
rationale: |
<what happened that taught us this>
evidence:
- date: YYYY-MM-DD
what: "<brief description of the event>"
The mechanics are simple. New instincts start at 0.5 confidence. global/ instincts apply everywhere, projects/<name>/ instincts apply to one project. A project pattern that shows up in two or more projects with confidence at or above 0.7 gets promoted to global. Anything that decays below 0.3 gets archived during maintenance, so a confirmed pattern climbs and a contradicted one drops.
The reason this beats a system prompt is that the rules come from real events, with the event attached. The rationale field records what happened that taught the lesson, the evidence list records each time it was confirmed, and the agent writes the rule itself so the next similar moment starts from what actually went wrong.
The context engine
Underneath all of this is a context engine that handles what happens when a conversation gets too long to fit. Instead of dropping old messages, it keeps a graph of summaries, so when context gets compacted the structure survives and you can expand back into the detail you need.
The layers do different jobs. The context engine preserves the shape of a single long conversation. Daily logs capture what happened across days. The promotion step keeps what matters. Instincts capture what to do about it next time.
The auto-router that came first
Before any of the memory work, I had already built a multi-agent setup: a router that dispatched a task to a specialist based on the work type. Docker specialist, network, security, media stack, monitoring, backup and recovery, debugger, deployment, docs, a PM role, a reviewer, each with its own focus and its own checks.
I am careful about how I describe this now. It was built on top of Claude, not a model trained from scratch, and plenty of people were doing agent orchestration around the same time. I did not invent the idea. I built my own because a single general assistant kept losing the thread on a homelab with 34-plus services, where the right move for a network problem and the right move for a storage problem are nothing alike.
That router shaped how the operator's roles work today: an orchestrator that hands off to a coder, a security role, a researcher. The lineage is real even if the bragging rights are not.
Making it reusable
The goal is to package the memory system, the instinct engine, and the role architecture into something one command can scaffold, so someone else gets a comparable operator without rebuilding the plumbing by hand.
The scaffold is the cheap part. The value shows up after install, in what the agent learns running your infrastructure. Your instincts will not be my instincts. Your durable facts will not be mine. The infrastructure just gives the agent a place to put what it learns, and a reason to learn it once.
What I learned
- Vector search is search, not memory. Memory is structured, curated, and intentional.
- Agents should write their own rules. A rule earned from a real failure, with the failure attached, survives a refactor of the system prompt. A rule typed into the prompt does not.
- Daily logs are cheap insurance. You never know which detail matters until the day you need it, and by then the only copy is the one you wrote down.
- Confidence scores matter. A proven 0.9 rule and a fresh 0.5 guess should not carry the same weight, and decay keeps the directory from filling up with rules that were wrong.
- Memory infrastructure is unglamorous. It is also the difference between a chatbot and something that remembers what happened yesterday.