Open models wishlist

The ability have a rag style internal log of tool calls so it has a fact collection. Things that are infinitely annoying is when you have a get-schema tool and it pulls table names. You don’t want add it as context and tokens but it also isn’t static. I would be nice if facts are accumulated and added as weighted responses. It doesn’t need to know the entire codebase etc it just needs to be able to build a map of facts For instance we make a client gui but context knows that when we talk about clients we are referring to a clients table as well. It shouldn’t need to do the tool call everytime because it should be accumulating self verified facts and then filing it. If we have a function called main and it deals with a window. Or there’s a function called main_window. And then we’re working on another file and refer to a visual aspect as Update the main window. Vs update the main windows function vs update the window in the main function because of the lack of know facts it doesn’t necessarily figure it out. So effectively I want a way that what it learns from tool calls to be come more like immediate referenced RAG style embeddings. Currently we do all this via agents etc but it seems to me that fine tuning / rag / context /function calling are not being trained into the models so to speak. If we train an llm with say CRUD for its own file system in ram drive as a possible option which adds to context but it can do its own function calls for access to “facts” about its current session. It’s a bit like and audit trail reporting results then ragging the results but in ramdrive. This would all effectively be able to be collected and added as fine tuning as needed but removes the need for constantly repudiating rag collections and context. Rag has index for projects and the memory files for each function call result with a chronology. That is where it does all its internal workings and build on it. It would seem to me if we can impart a way to factualize using weights based on database names/files functions etc in a ramdrive then we don’t need context to be constantly hit. Same as when 01 looks like it’s agents stuff on the fly. 🪰 f one of theirs steps was review internal knowledge about the database and previous chats and that was done via ramdrive crud inside the model then you will fix token burning. We play know that passing tokens back and forth with say the API or doco vs a websearch of the doco vs being given just the function example from a doco all get the right result but with horrendously different token values. Currently agents do this work and it’s fine but by bringing crud for a filesystem in ram the system gets smarter and more accurate. Ragging csv files is pretty destructive. Functioncalling SQL for data you have already seen just to get a table name again in a longer task in say cline can be like 200K tokens when it ran or created the schema itself in the previous sessions. Self documenting facts vs creativity is important. It should understand physics as fact rather than having flat earth vs round earth as being a debate based on how much it was takes about. In the same way we want functioncalling to do actual math because llms and numbers. we also need a way for it to learn the environment and keep its mind palace logical not just collections of words that may or may not have relevance.