The AI agent memory market hit $6.27 billion this year. Mem0 wants $249/month for graph features. Zep killed their self-hosted Community Edition and moved to cloud-only credit billing. Letta charges per agent. Everyone wants you in their cloud, topping up “memory credits.”

Here’s the question nobody asks before signing up: what happens to your memory when you stop paying?

Your accumulated context is everything that makes you unique. Every entity, every learned preference, every relationship your agent built over months, lives inside their system. That’s now your switching cost. Your memory is what makes your agent useful instead of starting from zero. And it’s locked inside someone else’s billing page.

Meanwhile: PostgreSQL is free. pgvector is a free extension. Your AI subscription already includes conversational memory. Markdown files cost nothing. Cron jobs are free locally. My total infrastructure cost is $0–7/month vs. $249–475/month for a managed framework. Here’s what actually works.

The 5-Layer Architecture

I landed on five layers. Each one handles a different type of recall. No single layer replaces any other.

Vigil Harbor's layered memory architecture for AI agents — five layers from conversational context to automated maintenance, all self-hosted with no vendor lock-in.

Layer 1: Conversational Context — Cost: $0 Session state, recent exchanges, preferences. This is Claude memory, ChatGPT memory, your system prompt. Already included in your AI subscription. Good for: “What did we just discuss?”

Layer 2: Structured Operational Memory — Cost: $0–7/month Entities, relationships, facts, events. I use PostgreSQL + pgvector. It handles structured queries, vector similarity search, and full-text search in one system. Expose it via MCP (Model Context Protocol) with namespace isolation per user/client. Graph edges for relationships. Good for: “What do we know about this customer?”

Don’t freak out; Claude can build one of these in one shot. It’s like the Skyrim of databases. Endlessly moddable and battle-tested. You don’t need anything fancy. The trick is tailoring it to what you need. I run about 10 MCP tools in \~2K lines of TypeScript: one for semantic search, one for structured filtered retrieval, one for graph edge navigation, one for upserts, etc. Just upload this post, it’ll ask some questions and build it for you.

Layer 3: Project & Task Knowledge — Cost: $0 Sprint status, decisions, blockers, ownership. Your existing task tracker (Plane, Linear, Jira) exposed via MCP or API. Don’t duplicate this into your memory database — it already lives somewhere. Just give your agent access. Good for: “What’s the status of this project?”

Layer 4: Institutional Knowledge — Cost: $0 Architecture decisions, conventions, file maps, SOPs. Wiki pages, repo markdown, Notion — whatever you already use. The key discipline: update after every merge and milestone. This is where your agent learns how your system works, not just what’s in it. Good for: “How does this work?”

Layer 5: Memory Maintenance — Cost: $0 Deduplication, conflict resolution, staleness detection, promotion/demotion. This is the hard part. Not the database. I use an agent cron job for daily linting and audit reports, then a second agent picks those reports up and operates on them. Two-job file-based handoff always. Research writes to disk, delivery reads from disk. It’s not perfect, but it’s working.

Without active maintenance, every memory system degrades within weeks regardless of how sophisticated its retrieval is. The managed frameworks mostly handle this poorly; Mem0’s implicit preference accuracy benchmarks at 30–45% on behavioral inference. “Intelligent forgetting” in most frameworks is just TTL expiration or recency pruning. Neither understands domain relevance, your specific knowledge you want to keep regardless of the policy.

What You’re Actually Paying For

Strip away the branding and pricing tiers, and every memory framework sells you four things:

1. A database with vector search. PostgreSQL + pgvector does this for free with ACID guarantees.
2. A retrieval layer. “Intelligent retrieval” is mostly hype. A well-structured pgvector similarity query gets you 80% of the way. The remaining 20% you might want, temporal reasoning, graph traversal, multi-hop, this only matters for specific use cases and most agents don’t need $249/month of it.
3. An extraction pipeline. Genuinely useful, but it’s an LLM call with a structured output prompt. You can build entity extraction in an afternoon. That’s not $249/month of engineering.
4. Lifecycle management. This is what they should charge for, because it’s the hardest to get right. But ironically, most frameworks do it badly.

The Data Portability Test

Before you commit to any memory system, ask three diagnostic questions:

1. Can I export everything in a standard format tonight?
2. Does it still work if the vendor disappears tomorrow?
3. Can I move it to a different system without rebuilding from scratch?

PostgreSQL passes all three. Markdown files pass all three. Your task tracker passes all three. Most managed frameworks fail at least one.

Honest Caveats

• This does take some engineering time upfront, easier with a coding agent of choice. If you’re a solo developer who just needs something working today, Cognee (open source, local-first, graph at every tier — genuinely good) might be the right starting point.
• The maintenance layer (Layer 5) is genuinely hard. I’m still iterating on mine. There’s no silver bullet for conflict resolution and decay.
• If you need enterprise compliance (SOC 2, HIPAA), a managed platform gives you those checkboxes faster than self-hosting.

Don’t build in someone else’s system just to accumulate into switching costs later. The most valuable asset for AI agents is your accumulated operational context; it directly effects their ability to work with “make no mistakes”. They’ll already know which mistakes you’re talking about. Take the time to build your own brain so no one can take it away from you.

Total infrastructure cost: $0–7/month. You own every byte of your data.

Happy to answer questions about the implementation. We open-sourced our MCP security layer (Drawbridge) and might do the same with parts of the memory tooling.