Why don’t local LLMs have memory ?
Posted by PumpkinNarrow6339@reddit | LocalLLaMA | View on Reddit | 21 comments
I’ve been using local models like Gemma 4 and a few others directly on my phone.
One thing I noticed is that there’s basically no real “memory” feature.
Like with ChatGPT or other hosted AI tools, they can remember context across conversations, sometimes even user preferences or ongoing projects. But with local models, every session feels stateless. Once it’s gone, it’s gone.
So I’m curious:
> Is there any proper way to add memory to local LLMs?
>Are people building custom memory layers for this?
>How do you handle long-term context or project continuity locally?
Would love to know how others are solving this.
Slight_Confection_66@reddit
Good question. Most local apps just don't implement it. You'd need to store conversation history in a local DB and inject relevant past messages into the context window. I built episodic memory into MBS Workbench — it indexes chats as embeddings and pulls top matches automatically. Works offline
PumpkinNarrow6339@reddit (OP)
Can u share 🙏
Left-Builder9483@reddit
https://chromewebstore.google.com/detail/lisa-core-ai-memory-libra/dmgnookddagimdcggdlbjmaobmoofhbj
This AI Memory extension may help if you use Chrome
Left-Builder9483@reddit
Use conversation exports with https://chromewebstore.google.com/detail/lisa-core-ai-memory-libra/dmgnookddagimdcggdlbjmaobmoofhbj
Savantskie1@reddit
I built my own memory system that has short term and long term storage. It also ingests chats as long term memory. Everything is tied back to those chats so if the model has to, it can search a chat by the chat logs. Short term memories are served on each model response. It’s not really suitable for small models and I’ve made it for models that have long context windows. It’s in my GitHub repository at savantskie under persistent-ai-memory. I didn’t link to it because I didn’t want to get into any trouble.
PumpkinNarrow6339@reddit (OP)
Thankyou
New_Dentist6983@reddit
There are tools like mem0 or screenpipe that helps with giving memory to AI, but fundementally LLM don't have memory in their architecture
iits-Shaz@reddit
Everyone's covered the "why" well — memory is a wrapper feature, not a model feature. Let me share a concrete approach that works well for on-device/local setups specifically, since you're running on a phone:
File-based memory with a flat index injected into the system prompt.
The idea (similar to what u/Dependent_Lunch7356 described with markdown files): store notes as individual markdown files on-device. Each note has a title, tags, and content. At the start of each conversation, you build a compact index of all notes — just titles + tags + first line — and inject it into the system prompt.
When the model sees "Flight Info — travel — April 15th Delta DL1234" in its system prompt, it knows that information exists. If the user asks about their flight, the model can request the full note.
Why this works better than RAG for small-scale on-device memory:
The tradeoff u/Dependent_Lunch7356 mentioned is real — re-reading context is expensive. But on-device the "cost" is just tokens in the context window, not API dollars. The flat index approach keeps it cheap because you're only injecting the index (~1-2KB), not the full notes, into every conversation.
I'm building exactly this for a React Native SDK right now. The hard part isn't the memory system itself — it's teaching the model when to save vs. retrieve. A good system prompt that says "when the user tells you to remember something, save it as a note" gets you 80% of the way there.
Dependent_Lunch7356@reddit
appreciate the breakdown and the tag. this is exactly right — the flat index approach is what works at small scale. my setup is cloud-based anthropic API so the cost is real dollars, not just context tokens, which is what made me start tracking it. the 83% cache write cost i mentioned is basically the price of that re-reading step on every single turn. on-device you eat it in latency instead of dollars but the tradeoff is the same — memory is the most expensive thing an agent does. interested to see your react native SDK. the "when to save vs retrieve" problem is where most setups break.
iits-Shaz@reddit
Exactly — the cost structure just shifts. On-device you pay in context tokens instead of dollars, but the constraint is tighter (4-8K phone context vs basically unlimited on Claude). So on-device memory has to be curated, not deep. Still designing that part for v0.2 — the save vs retrieve heuristic is where I expect most of the iteration. Will share when it's ready.
Dependent_Lunch7356@reddit
that 4-8k constraint is interesting — forces better design honestly. curated memory that fits in 4k is probably more useful than dumping everything into 200k context. looking forward to seeing v0.2.
ApexDigitalHQ@reddit
I haven't played with this on mobiles yet but I am playing around with obsidian vaults and making "wikis" as referable memory sources for my local projects. The results have been pretty good so far but not anything groundbreaking.
PumpkinNarrow6339@reddit (OP)
Cool, any resource?
ApexDigitalHQ@reddit
This video explains it better than I can. I just keep all of my vaults in a shared space so they can be referred back to as necessary. https://youtu.be/zVEb19AwkqM?si=afLE1_TSiLJomtho
Dependent_Lunch7356@reddit
i run an agent on claude through openclaw — it uses markdown files as memory. every session it reads files to remember who i am, what we've been working on, what decisions we've made. works but it's expensive. tracked my costs for 27 days and 83% of the bill was the agent re-reading its own context. memory isn't free — it's the biggest cost driver in the whole system.
PumpkinNarrow6339@reddit (OP)
True
Dependent_Lunch7356@reddit
to answer your actual question — mine stores memory in markdown files. daily logs, long-term memory, user profile, identity doc. i configured the agent's instructions to read those files at the start of every session and update them as conversations happen. the framework openclaw handles injecting workspace files into context, but the memory structure and read/write behavior is something i set up myself. the model resets every session — the files are what make it persistent.
Ulterior-Motive_@reddit
It's entirely dependent on the client you use. LLM memory is kind of a fiction, in the sense that the entire chat history is being sent to the model each time you send a new message (it might get cached for performance reasons, but for all intents and purposes it's inaccessible to different chat sessions). I know Open Webui lets you directly reference other chats.
ResponsibleTruck4717@reddit
As far as we know no model currently have memory, once the model is trained you can't add new data to it without further training.
What you can do is workaround, giving it context, like rag does.
Lets say you had conversation about chicken recipe, once you close it the model will not remember it.
But if you save the conversation, and the next time you talk to the model you provide it with the highlight / history of latest conversation, it will give you the illusion it remember your conversation.
I wrote you very simplified explanation but the challenge is to manage large context, and there are few tricks and method to improve it.
x11iyu@reddit
no LLMs have memory; the online ones that seem like they do simply pull those content you notice it 'remembers' into the llm's context behind the scenes
broadly speaking searching for RAG may help you see more info on this
VoiceApprehensive893@reddit
find a client that has a memory feature
"memory" is summarisation not a part of the model