Creating the brain behind dumb models

Posted by ChristopherLyon@reddit | LocalLLaMA | View on Reddit | 119 comments

I've been fascinated by model intelligence enhancement and trying to deploy super tiny models like gemma3:270m in niche domains with high levels of success...

My latest implementation is a "community nested" relational graph knowledgebase pipeline that gives both top down context on knowledge sub-domains, but also a traditional bottom-up search (essentially regular semantic embedding cosine similarity) with a traversal mechanism to grab context from nodes that are not semantically similar but still referentially linked. Turns out there is a LOT of context that does not get picked up through regular embedding based RAG.

I created a quick front-end with nextjs and threejs to visualize how my knowledge base hangs together, and to quickly identify if I had a high level of overall coherence (i.e. number of isolated/disconnected clusters) and to get a better feeling for what context the LLM loads into memory for any given user query in real time (I'm a visual learner)

The KB you can see in the video is from a single 160 page PDF on Industrial Design, taking you anywhere from notable people, material science to manufacturing techniques. I was pleasantly surprised to see that the node for "ergonomics" was by far the most linked and overall strongly referenced in the corpus - essentially linking the "human factor" to some significant contribution to great product design.

If anyone hasn't gotten into graph based retrieval augmented generation I found the best resource and starter to be from Microsoft: https://github.com/microsoft/graphrag

^ pip install graphrag and use the init and index commands to create your first graph in minutes.

Anyone else been in my shoes and already know what the NEXT step will be? Let me know.

It's 2 am so a quick video shot on my mobile is all I have right now, but I can't sleep thinking about this so thought I'd post what I have. I need to work some more on it and add the local LLM interface for querying the KB through the front end, but I don't mind open sourcing it if anyone is interested.

[-]

danieltkessler@reddit

Okay, so OP... I think we're all going to need a full tutorial on how you made that exceptionally gorgeous visual.

[-]

FoxB1t3@reddit

I upvoted before reading just for the visual effect lol.

[-]

Infamous_Explorer_71@reddit

I found this repo the other day and was thinking, if they could add graphing to it memory storage. This could be next evolution https://github.com/rairesearch/teaser-repo

[-]

redlikeazebra@reddit

Hey I can't find a link to the repo in the comments. Is it open yet?

[-]

ChristopherLyon@reddit (OP)

https://github.com/ChristopherLyon/graphrag-workbench/tree/v0.1.0-alpha.1

I just got it done today :D

[-]

AntisocialByChoice9@reddit

!remindme 3 days

[-]

teachersecret@reddit

The visual is gorgeous.

[-]

justV_2077@reddit

Yes also I like the color palette of blue and red. It reminds me of the cyberspace from Cyberpunk 2077.

https://cdnb.artstation.com/p/assets/images/images/041/217/811/large/peter-ankermann-cyberspacecharacter-08.jpg?1631093284

[-]

Embarrassed-Farm-594@reddit

We have a CB 2077 stan here.

[-]

Neither-Phone-7264@reddit

r/usernamechecksout?

[-]

PrestigiousBet9342@reddit

This looks AWESOME !!!

[-]

UnreasonableEconomy@reddit

One thing I've learned from this sub, is that if you want to wow some noobs, just orbit a 3d FDG XD

[-]

Putrid_Speed_5138@reddit

Forgive my ignorance but what is 3D FDG?

[-]

UnreasonableEconomy@reddit

3d force directed graph

[-]

Putrid_Speed_5138@reddit

Ah, OK, thank you.

[-]

No_Sandwich_9143@reddit

You got me

[-]

Classic-Rise4742@reddit

!remindme 36hours

[-]

Classic-Rise4742@reddit

!remindme 24hours

[-]

mortyspace@reddit

Looks cool, any practical use case? or just for research?

[-]

ChristopherLyon@reddit (OP)

This has really significant practical use cases for my day job doing subsea robotics. But this project is also research for my new inferance platform -> www.slmwiki.vercel.app

[-]

mortyspace@reddit

Could you give examples how you use in subsea robotics, really interesting topic, thanks 🙏

[-]

ChristopherLyon@reddit (OP)

Graph for me will come in clutch for things like parsing increadibly lengthy and complicated troubleshooting manuals. With relationship discovery being able to troubleshoot a 60VDC power supply and know WHAT it supplier, WHERE it's mounted, what supplies IT power ect, all in LLM context is a game changer. Worst case we can get fines 500kUSD per DAY we are ok downtime, so using something like this to get things online faster is an incredible financial win.

[-]

mortyspace@reddit

If I got you right, you ask llms to check docs/per specific model chain? I heard it could hallucinate pretty well on lengthy stuff, did you experienced this cases?

[-]

ChristopherLyon@reddit (OP)

That's what the graphrag helps with, to stop hallucinations by grounding the model with citable context at a low temperature.

[-]

mortyspace@reddit

Got it, interesting, curious how do you measure or detect hallucinations like you scan doc and then if it's citates wrong you have some graph visual detections. What % for this model you got so far like in avg before taking action to align behavior with low temp?

[-]

ChristopherLyon@reddit (OP)

So far in all my tests I've gotten 0% hallucinations. All hard refernacable material like part numbers, values, standards, measurements ect I've been able to look up in the source material and find pretty much verbatim since the graphrag system is so good at finding context, the LLM nearly lifts a finger. I'm also using a local thinking model which really helps give good answers based on the RAG info. (Qwen3 and gpt-oss:20b

[-]

Artistic_Okra7288@reddit

I think the best way to build external knowledge for the LLMs is by something like GraphRAG. A knowledge graph database is built on relationships and definitely seems like it should be much better than straight RAG. Think like valve specs, you'd have to store too much information with each vector to do the same thing the knowledge graph would do (provides context, relationship), so with straight RAG, you might get the wrong specs because that kind of context, they all almost look identical, so it's a toss up which one the LLM looks at (top_k results), you could get all top_k results as the specs for all wrong valves. But if you walk the knowledge graph, you'll get the exact specs to the exact valve you are talking about every time (assuming your knowledge graph is built without errors). So the hardest thing at this point is keeping the KG in sync with current data, and that's a huge pain of rebuilding the graph from scratch to refresh it.

[-]

mortyspace@reddit

wow 0%, feels like you solved world problem with hallucination, impressive!

[-]

ChristopherLyon@reddit (OP)

Red my other comment, this is what the system looks likw

[-]

Weary-Wing-6806@reddit

This is really cool. Graph-based RAG feels like the next logical step as embeddings miss too much context, and graphs pull in those connections you’d never catch otherwise. Love the visual angle too, it makes the whole thing click. Def open source it if you can, we'd jump in fast!

[-]

ISoulSeekerI@reddit

I like it, does it run okay on Ubuntu or Linux machines?

[-]

ChristopherLyon@reddit (OP)

It runs everywhere 👍

[-]

ISoulSeekerI@reddit

Fantastic, I’ll check it out.

[-]

Simple_Paper_4526@reddit

jarvis, zoom in

[-]

NebulaNinja182@reddit

!RemindMe 1 month

[-]

RemindMeBot@reddit

I will be messaging you in 1 month on 2025-10-01 17:43:24 UTC to remind you of this link

CLICK THIS LINK to send a PM to also be reminded and to reduce spam.

^(Parent commenter can ) ^(delete this message to hide from others.)

^(Info)	^(Custom)	^(Your Reminders)	^(Feedback)

[-]

Ok_Association_1884@reddit

this is mad cool dude! im on my 4th model, this one is foundation model and includes reasoning and multi-modal capabilities with recursive self education, not quite self improvement by RL standards as its insight/experience based on strict data driven.

What you are doing is how i generated lobe portions of my AI's digital brain. you can find an IBM training dataset, i cant find where i got it atm apologies but this repo references the IBM Framework hub https://github.com/IBM/powerai-transfer-learning, but it will provide primitives for your graphs foundation vector points if data validity isnt a requirement for the experiment. idk if that relative to you at all...

[-]

Innomen@reddit

Apologies for the cynicism but this seems like jargon salad to me. Does it lead to a chat box with hard drive (instead of vram) memory or not? I tried to do something like this too: https://github.com/Innomen/TextScape But I quickly realized it wasn't really working. The concepts sound nice but i just don't see any evidence they actually fit together. Even rag itself i'm skeptical about, like what's the difference between rag and common find in file search? I mean, i wanna be wrong here, i have a lot of papers i'd like to work with as a single project. ( https://philpeople.org/profiles/brandon-sergent/publications ) But every time i look into a tool it looks and sounds like this which also looks and sound like my textscape. Which i know (mine) is a mirage. I don't see any evidence that even the big 4 can do anything like this. Grok and gpt kinda try but again, what's the difference between rag and just searching normally. So ok the data can be kind retrieved but it's not being synthesized or absorbed. I know because i can ask the ai questions about my work and it gives nonsense answers unless i hold its hand. /sad rant sorry

[-]

Auto_Luke@reddit

Open sourcing it would be great. If not, can you point to an existing open source implementation that is the closest to your solution?

[-]

DeathShot7777@reddit

It is opensourced and I work on it whenever i get time juggling college and work 😮‍💨. Here is the repo: https://github.com/abhigyanpatwari/GitNexus

Pls star it if u found it interesting. Might help me convince my CTO to allot time for this even though it is personal project 🫠

[-]

consolecog@reddit

This is so cool! Could you dive into each element in the graph?

[-]

DeathShot7777@reddit

I m working on a Knowledge Graph generator from codebase. Runs completely client sided in the browser. The relations are generated using Tree-sitter to map out within file relations and external file import maps. Gives a Graph RAG agent on the side. Might be similar. It is still WIP ( working on parallel processing and graph db instance that also runs inside the browser )
https://github.com/abhigyanpatwari/GitNexus

Are u generating the relations with LLM or script?

[-]

n4il1k@reddit

i skimmed over your repo a bit, how do you build the dependency graph and do you only link function definitions? if so how does this perform with object oriented projects?

[-]

DeathShot7777@reddit

I have created this 4 pass system for the relations:

Pass 1: Structure Analysis: Scans all file and folder paths to build the basic file system hierarchy using CONTAINS relationships (e.g., Project -> Folder -> File). This pass does not read file content.

Pass 2: Definition Extraction & Caching: Uses tree-sitter to parse each source file into an Abstract Syntax Tree (AST). It analyzes this AST to find all functions and classes, linking them to their file with DEFINES relationships. The generated AST for each file is then cached.

Pass 3: Import Resolution: Analyzes the cached AST of each file to find import statements, creating IMPORTS relationships between files that depend on each other.

Pass 4: Call Resolution: Re-analyzes the cached AST for each function's body to identify where other functions are used, creating the final CALLS relationships between them.

Pls star the repo if it was interesting. Might help me convince my CTO to allot some time on this even though it's personal project 🫠

[-]

n4il1k@reddit

Do you also have a way of handling function definition and class definitions which exceed the context window of your embedding models?

[-]

DeathShot7777@reddit

Great question, you pointed out an edge case which might be possible in monolithic codebases with huge functions. I am not using an embeddings model or any LLM help at all, to generate the Graph so context window issue wont occur during KG creation. For retrieval by the Graph RAG it may stumble across a node containing the huge function, but considering LLM context windows are generally 128K and above normally it shouldnt happen. If there is a function that dont even fit in such huge context window, that codebase might be beyond me to try to understand LOL

[-]

ParamedicAble225@reddit

glad to see others playing with LLM's and trees.
https://github.com/taborgreat/creative-ideas/tree/dev-2.0

[-]

Local_Metal_4175@reddit

How do you handle knowledge db updates to avoid document staleness w/ graphrag? Are you just regenerating the graph each time?

[-]

onehitwonderos@reddit

Super interesting! Thanks for sharing. Open sourcing it would be great 🙏

[-]

SignificantPound6658@reddit

Love it, Thanks for the repo. I was about to dive into GRAG

[-]

horsethebandthemovie@reddit

What do the really small models do in your system?

[-]

Maxwell10206@reddit

This would be great for debugging LLMs I bet. If I remember correctly how LLMs think is still a mystery but maybe using a visual tool like yours it would become easier to understand how they work and how to improve them! Great job!

[-]

Girafferage@reddit

They don't think. They are just statistical models. You run them over and over and over through different weights for a bunch of epochs and when you get one that the results come out similar to how you expect in your training data, you call it a win.

I don't honestly think this would help with figuring out how the weights play together to get the desired result, since the weights don't contain information in that way like you can just see where any given piece connects. But the idea of something like this to try to open up the black box even a little is really cool.

[-]

colin_colout@reddit

This looks like RAG. Not visualizing the llm inference but creating a second "brain" to feed context and knowledge into smaller llms

[-]

M4rs14n0@reddit

More GraphRAG I'd say

[-]

Adventurous_Top8864@reddit

What are your hardware specs? Curious to know looking at how seamlessly it is working

[-]

ChristopherLyon@reddit (OP)

MacBook Pro M3 with 36gb RAM

Runs super smooth

[-]

Adventurous_Top8864@reddit

Thats definitely a powerful machine!!!

[-]

kylebrodeur@reddit

Man I’ve been trying to come up with a view like this.

[-]

Californicationing@reddit

This is game changing for visual thinkers, brilliant. Hope this tool reaches a lot of beautiful minds!

[-]

bio_risk@reddit

I see what you did there.

[-]

to_takeaway@reddit

Insanely cool. I'm a huge fan of graphRAG, using it daily in pet projects and at work too. I'm using the nano-graphrag wrapper for its simplicity (https://github.com/gusye1234/nano-graphrag).

I'm using neo4j browser to visualize this, but it would be awesome if you open sourced this.

[-]

ChristopherLyon@reddit (OP)

Yeah I need to try nano!

[-]

urmel42@reddit

Could you please describe how you created the dataset out of the pdf to train the LLM? Currently struggling at this...

[-]

ChristopherLyon@reddit (OP)

It's very very easy. I've spent ages making advanced systems that use LLMs to create perfect chunks ect, but tbh in this approach it's just .pdf -> .txt -> split into 1500 token chunks with 100 token overlap.

My source material is controlled because it company data, so I don't have to OCR dumb graphs or any of that hard stuff this time around.

[-]

w8nc4it@reddit

Very similar to lightrag, you can look into it here if you are interested: https://github.com/HKUDS/LightRAG, it's hybrid search mode also combines sematic search and graph based search. It uses an llm to tag (NER) both queries and stored content chunks for search and retrieval, which is what drives the graph based searching. It also has knowledge graph visualization.

[-]

raiffuvar@reddit

Light rag is great at first sight....but i hit limits with openai keys... I short: if it's work should be super great. But not for me.

[-]

w8nc4it@reddit

OpenRouter generally offers much higher rate limits for paid API access compared to OpenAI, especially with openais tiered rate limits system based on total spend. I personally use lightrag with openrouter or a local ollama model without having to mind any kind of rate limiting.

[-]

brownman19@reddit

Yes you need to define coherence metrics and isolate the "fields" that define those clusters.

You can essentially find the geometry and curvature of that feature cluster, optimize the curvature, reduce dimensionality (look up concepts like matryoshka reduction), and then start targeting context autonomously based on signals.

Other extension ideas (somewhat related) you can use from my repo:
1. Auto indexing on the fly as agents work to build that graph in realtime: https://github.com/wheattoast11/openrouter-deep-research-mcp/blob/main/src/utils/dbClient.js
2. Think of the clusters you are interested in including in the context, and try to log a state parameter for the environment to give to the agent as context that brings that cluster into the semantic retrieval more readily. Here's an example of the level of state management I include in agentic apps. What you see is basically the ability to "time travel" in any given session to any event on the app. A more extreme case but because the agent is aware of this feature and how it changes the app states, it is contextually aware of current state, the fact that we are rewinding to prior states. All the context retrieval is semantic and fully sliding window and intelligently parsed/self managed.

Think of the knowledge graph as a "home" and how you want to carve out the rooms in that home. Build your agent system's retrieval operations to anchor to those "rooms" as a concept so that they can retrieve and match on the right clusters during graph operations/retrievals.

https://i.redd.it/65bedxb5bfmf1.gif

[-]

NobleKale@reddit

It's 2 am so a quick video shot on my mobile is all I have right now, but I can't sleep thinking about this so thought I'd post what I have

You've done well, but you should probably get to sleep, yeah?

[-]

ChristopherLyon@reddit (OP)

Yeah... 🙏

[-]

Otherwise-Tip-8273@reddit

nice frontend

[-]

ChristopherLyon@reddit (OP)

It got a massive upgrade today. Didn't quite have enought time to publish but will try tomorrow! Thanks!

[-]

skinnyjoints@reddit

This is incredibly interesting to me but I am completely new to knowledge graphs so please correct me if I am wrong. My understanding of this is that you:

Create a map of sorts of all the interconnecting concepts and entities in a corpus and their relationships to each other
Each concept gets a node which has a semantic embedding. When queried, certain nodes are returned as semantically similar. You then branch out to the nodes that they are connected to.
You are left with a subsection of the knowledge graph consisting of the relevant nodes and how they are connected to other nodes. This is basically finding the concepts relevant to your query and how they are related to other concepts.
You then compile all this into text which you can supplement a small LLM with?

Is this interpretation correct?

[-]

raiffuvar@reddit

You have a node class. If you only have text, all you can do is ask a question and return answers similar to answer embeddings.

So, people do hacks: generate several questions from one question to get more answers, hoping their real answer will be in those results.

With a graph, you can get neighbors, which leads to more diverse results.

Obviously, it's only one example, but "similarity" is the wrong word here. In classic algos, you truly can compare nodes based on neighbors. But for RAG, you search the answer.

Although, if you find one embedding, look at neighbors to the parent class or to children or to functions where this class is called. You can't do this without graphs.

For books, it should be a little bit different, but it highly depends on the task.

[-]

CodeSchwert@reddit

I’ve just started getting into knowledge graphs for RAG too and that sounds about right. Deeplearning.ai has a short course on it with Neo4j that’s a pretty good intro.

[-]

OysterPickleSandwich@reddit

Link?

[-]

CodeSchwert@reddit

https://www.deeplearning.ai/short-courses/knowledge-graphs-rag/

[-]

anujagg@reddit

What exactly are you trying to achieve here? Can you pl explain in simple terms? Sorry for being a noob but not able to get this concept and how it could be used. Thanks.

[-]

ChristopherLyon@reddit (OP)

No worries! The simple version is that small models lack intelligence. This is a retrieval system that enhances the in-siti context of the LLM based on the context of the users question.

What makes it special is that it finds relational information over plain embedding search so you get a better understanding of influences/dependencies ect.

[-]

DinoAmino@reddit

Right? No code. No repo. And people fall all over themselves. Lol. The glowing praises. The outsized number of upvotes ... and the awards?!? No one gives out awards in this sub. There are no regulars commenting on this one. This is just another bot fueled post of little value. I guess we will be seeing this stuff more often because there is a locallama discord now and people want meaningless flair.

[-]

ChristopherLyon@reddit (OP)

I'm giving out the rewards? It's like 4 dollars worth??

What the actuall hell are you on about? I linked the code, the repo is comming and I'm sharing my research and resources with other like-minded enthusiasts.

If you read the post or looked at my profile you'd see how genuine it all is.

The amount of support I've gotten in contrast to your level of distain/envy shows just how out of touch you are, and I invite you to do better 🤏

[-]

AppealThink1733@reddit

It looks a lot like Obsidian. It would be great to have an implementation where the AI could search the repository and information saved in Obsidian and respond to the user on another platform. I found it interesting.

[-]

thejoyofcraig@reddit

https://github.com/MarkusPfundstein/mcp-obsidian

Where there is an MCP there is a way…

[-]

raiffuvar@reddit

Evening won't be wasted thx. To you.

[-]

aaronsb@reddit

For variety, here's another obsidian plugin that presents graph traversal as well.

[-]

AppealThink1733@reddit

Thank you for sharing, it will be very useful!

[-]

raiffuvar@reddit

Hope it's all models and you need 3 trillions of those to create brain.

[-]

Someoneoldbutnew@reddit

the next step would be open sourcing this badass.

for reals, your next step is to visualize the linkage for any given LLM query. I've build graph systems before, you got the hard part out of the way, now it's time to USE it.

[-]

ChristopherLyon@reddit (OP)

Just finished it today. Didn't get time to actually deploy the repo, but it's comming - and with significant updates.

[-]

tigerjjw53@reddit

I don't understand what this is, but it looks like what people in 80's imagined computers would look like in 2025

[-]

AlbanySteamedHams@reddit

2015: We were supposed to have hover boards by now. I feel cheated.

2025: This is the techno dystopia I was promised as a child. My feelings are mixed.

[-]

Unlucky-Cup1043@reddit

I am looking for a kind of llm visualization like this. Anyone has a Video from this ?

[-]

AlexandreFSR@reddit

Yes pls open-source! Amazing work 👏👏

[-]

SmartEntertainer6229@reddit

Tony stark is that you

[-]

nostriluu@reddit

Many of my exploratory projects have such a graph, though this is a particularly nice one. They are dazzling, and can be used for debugging and to visually check cohesion, but they become very jumbly when they are heterarchical rather than hierarchical. Every once in a while some project features a graph and people go wow, going back to Flash visualizations of Wordnet. A lot of web companies feature such a graph as a background on their web pages, but they're missing the edges.

I'd hope most people have heard of the 'semantic web,' it proposes that the entire Web be a type of graph. Here's a not very sexy graph https://lod-cloud.net/# Roughly, the semantic web uses symbolic logic and entailment, whereas neural systems use probability/proximity. Some people think neuro-symbolic AI will become important since symbolic AI can be precise.

[-]