200+ pages of Hugging Face secrets on how to train an LLM

>My real issue with context graph approaches is just how do you extract entities-relationships from a real life document (or a memory observation) without a significant amount of "normalizing" which can in itself lead to loss of meaning. And without the "near by graph" addition. pure vector search is hit-and-miss. I think the difference between what I'm doing and what you're thinking of, is that I'm not really constructing a full knowledge graph or doing reasoning across graphs. The context graphs for a documents are literally just nodes for the document, whatever chapters or sections, the paragraphs, the sentences and tables and images or whatever. There's an embedding for each node, and a bit of meta-data. At this point, the LLM is almost totally ignorant of the graph (though I am playing with adding extra instructions to the system prompt), The wrapper pulls what it thinks is needed and prepends that to the prompt. The search is generally very good, at least as far as I've tested it, because I've got multiple vectors for each document, not just one high-level overview. There's definitely overhead, but the overhead lives outside the model in a much cheaper environment. By "make fast decisions about what the user wants" I don't mean the small model makes any decision about what documents are relevant or not, I mean, literally just interpreting what is is that the user wants based on the prompt, so you decide if the request needs a simple document retrieval, or if it needs synthesis of multiple documents, or if they're just referring to an earlier point in the conversation. You can use the smaller model and the wrapper to guide the preprocessing you do, and manage the bigger LLM's responses. For instance if the user has a database of 1000 books and wants to know which books involve [concept], then that's a question about meta-data, and you don't need to bring all 1000 books into context, you can search the embeddings for ones near [concept], and you can potentially get an answer without having to bring any books into context. The hierarchical embeddings let you have that granularity. If the user asks a more nuanced question about something per-book, you can internally run 1 inferences on each book, and aggregate the answers; You don't need all 1000 books in context at the same time. If the user wants a comparative analysis, then you might need multiple books in the context at one time, but also, you've already got multi-layered embeddings for every book pre-computed; You're already halfway there, and once again you can run multiple inferences can aggregate the answers. So again, this kind of context management doesn't actually make the model smarter in an absolute sense or increase the model's capabilities, but you can make it "feel" like more, because on the backend you're leveraging work that you've already done, and can be running multiple inferences without the user having to manually do it. My thing was started before the Stanford "In-the-flow" AI model paper was published, but there are some overlapping ideas that are relevant: https://agentflow.stanford.edu/ Their thing is basically an even more advanced, tool-aware set of models that need additional training, but the idea of a manager model making plans and overseeing the use of tools and being a layer that manages the LLM, is conceptually pretty close.