What is next for local LLM and AI?

Posted by GodComplecs@reddit | LocalLLaMA | View on Reddit | 26 comments

What is the next frontier for local llm and local AI? Myself I've been developing a system around the llm, using it only as an language interface, but being controlled by FEP principles, adding error prediciton mechanisms. Long term is to create digital NPCs for games and worlds that can autonomously create memories and their own story lines, controlled by emotions that are based on emulated physics. I still have a huge problem with what their interests would be that can form the long term memories, but solved the information problem (LLMs posess vast amounts of data, how do NPCs learn new data) What have you guys been working on as of late? Even the little things are interesting.

Reply to Post

26 Comments

[-]

--Spaci--@reddit

The next froniter for local llms isint llms, llms are a stepping stone to real AI

[-]

Hero_Of_Shadows@reddit

I've been doing something in the same general area, I love world generation I want to generate worlds, countries, npc's and dynamically build story lines from that. Like say the player enters a village and the village has 20 married couples 50 unmarried people of marriageable age you would randomly get a quest from one of the men to get some quest item from a dungeon so that he can offer it to one of the marriageable women. Or a city with a higher than normal amount of NPCs associated with the thieves guild this spawns quest to drive them out or etc. And the reverse a thieves guild in one city might send you to a nearby hamlet to turn the hunter into a poacher that will get them furs or the trader into an fence. I want a Daggerfall style sandbox but really generated. An npc would load into the LLM all the info (guild, political allegiances, character traits, past missions) and offer up dialog.

[-]

JohnBooty@reddit

I’ve been thinking about this too a lot, been designing it in my head for ages… the “LLM as DM” idea so many have had. The database will be prefilled with a lot of hand-written world details, rules, and characters. AI will then modify that and expand on it. Database will then store conversation logs and game state like character health, relationships, motivations, etc. It will be stored in a structured way so the game engine can just pull in the bits it needs, keeping the context window small. I’m stumped at how to represent a whole sandbox world with emergent quests and storylines. So I think what I’ll do is, start with a SINGLE scenario. Like the equivalent of a one-shot D&D campaign. A single “retrieve the thing from the dungeon” style quest. What interests me most is emergent behavior. The idea of defining a rich sandbox world and seeing interesting behavior that I never would have thought of. Can I define an interesting world and characters and let the LLM build upon it???

[-]

Objective-Error1223@reddit

Is there any game design platforms that work well for this? I’ve been using mine to code a MUD I’ve always wanted to play when I was younger, would love to know if I could upgrade the graphical experiences. 😂

[-]

Hero_Of_Shadows@reddit

I haven't found one yet

[-]

GodComplecs@reddit (OP)

Hmm this would be possible ofc, very agent heavy, but lots of stuf fin database / context. Are you trying to run it live or pregenerate stuff? Pregen for most stuff would be more efficient.

[-]

Hero_Of_Shadows@reddit

Pregen, quests would be live.

[-]

Time_Cat_5212@reddit

I too had the idea of making WoW NPCs (and playerbots) powered by LLMs. Would love this

[-]

WhopperitoJr@reddit

You might be interested in taking a look at what I do with my [plugin for Unreal Engine](https://swamprabbitlabs.com/personica/) that provides a data framework for injecting dynamic context into prompts. NPCs write and store long-term memories which are then searched via TF-IDF comparison whenever the player submits a new input. If a memory has a high matching score, it gets injected into the prompt as a relevant point to remember. One thing that works well for me if you are building a memory system is to add decay functionality. For instance, you could simply ask the LLM to provide an importance score from 0-1.0 when generating a memory, then store that float and decay it over time, either linearly or exponentially depending on how you want to design a game. The next true “frontier” imo is basically using techniques similar to MTP to confirm reasoning and adherence to game lore. I saw a recent method posted on here (called CoVE or something?) that I have been meaning to look more into.

[-]

GodComplecs@reddit (OP)

[https://arxiv.org/html/2506.19993v1](https://arxiv.org/html/2506.19993v1) That one right?

[-]

WhopperitoJr@reddit

Yes, this may not be directly useful in the case of games, but a conceptually similar implementation of finetuning could reduce hallucination and improve performance.

[-]

GodComplecs@reddit (OP)

Yes but using it eg. for generated quest chains based on moral choices would be pretty cool. Perfect recommendations seem to be an very untapped region, but a little locked down since you need static values for everything.

[-]

oldschooldaw@reddit

You have my attention with a wow scrot. I wonder how far I can push injecting a bridge between gedonia and a local model to supplement the terrible NPC dialogue. I wonder how quickly it’ll get stale. I wonder if it’ll still make me regret spending that $5 on that terrible game.

[-]

GodComplecs@reddit (OP)

If the base is terrible it's hard to improve on with an llm on the first pass, you need to focus on a rewrite in the style you like first.

[-]

FatheredPuma81@reddit

Next I hope companies train their LLMs to get and understand coordinates in uploaded images. Then I can task them with playing BTD5 using a program that lets them move the mouse.

[-]

DoorStuckSickDuck@reddit

Bounding box calculation has been a thing for a while now

[-]

optimisticalish@reddit

We still need local easy-to-use inline translation of bitmapped images. Qwen 3.5 can OCR and translate an image in an instant, but it can't also erase the foreign language words and seamlessly replace them with English in the same font.

[-]

ThisGonBHard@reddit

While Flux Klein is not there, image models are getting there. Proprietary ones are already capable of that, even for weird ones/angles, but open models lag behing. You can make a pipeline that automatically splits the image in panels, translates it, send it to an edit model, and then stiches it back together. It wont be simply, but it is for sure possible.

[-]

ComplexType568@reddit

Pipelines like this exist, I remember someone posting a manga translator. Ofc it can't do angles (I think) but a POC is already here

[-]

More-Curious816@reddit

i remember someone did that. not the exact font but what you described. can't remember here, Twitter or Discord, will report later.

[-]

GodComplecs@reddit (OP)

Seems very last gen, cant this be implemented with fine tuning and a pipeline?

[-]

fsociety10@reddit

I myself am in the same boat. But it's all about build vs buy decision. For personal use, it's okay but for building a product I am afraid GPUs costs are higher for low traffic. This is without considering the limited capabilities in audio/vision

[-]

GodComplecs@reddit (OP)

It depends on what you are doing, edge models aka small llms have gotten a lot better, so you could offload some of the gpu tasks to the player or user.

[-]

fsociety10@reddit

I agree it depends on the use case. I am building a product to help optimize the burning AI costs and everyone doesn't have the budget or time to selfhost and maintain a localllm. That's where the managed solutions work. Simple build vs buy analysis.

[-]

ttkciar@reddit

One of my early LLM projects was generating "flavor text" for a fantasy setting (Krangath, in Gehenna), but those early models just weren't up to the task. The models have gotten better and my own prompting skills have gotten better, so it would be fun to get back to it. In particular the context lengths are a lot longer nowadays (256K tokens vs 8K tokens). Perhaps I could simply pre-load context with all of the setting and existing flavor context I have, and ask Gemma4 or K2-V2 to generate more?

[-]

GodComplecs@reddit (OP)

Yes you can make an outline, then an conductor to steer new characters. You can make a custom software with opencode which through then you can make an automatic looping system to create these tasks. It can dynamically load the context needed per character, if you make it so.