What software do you use to interact with local large language models and why?

I use llama.cpp. It's neatly self-contained, without many external dependencies (none, if inferring on CPU), and is laid out as a "do everything" project -- inference, training, fine-tuning, whatever. It is its own complete ecosystem in microcosm. I am mindful of the next AI Winter, and that many of these open source projects might be abandoned in its wake. llama.cpp is simple and straightforward enough that I could continue to develop and maintain it myself indefinitely, if need be, as I have a few other projects. I do use RAG, and have developed my own system which is quite a bit different from how RAG is normally implemented. Instead of chunking data into fixed-sized segments, transforming it into embeddings, and putting it into a vectordb, I am organizing it into variable-sized documents and indexing it with Lucy (a conventional text search engine) as text. My RAG utility uses the prompt to perform a full-text search and retrieve the best-matching documents, and then uses a summarizer based on nltk to condense the documents into just the text most relevant to the prompt until it can fit within the model's context. Then the summarized text is prepended to the prompt (still as text) and the inference stack vectorizes it at inference time. There are some disadvantages to this -- it is slower than conventional RAG, and full-text search does not always do as good of a job finding relevant content as semantic search. There are some advantages, too, though -- inference begins with context more densely filled with most prompt-relevant content, and I can switch between different inference models without revectorizing my entire database. It's convenient to start with a 3B model for quickly iterating through several search/infer cycles, and then switch up to an 11B to repeat a prompt for which I want higher quality inference. I'm also developing a new feature for llama.cpp, self-mixing, which accomplishes at inference time the same effect as a self-merge, with a `--layer-order` parameter directing it to infer with different sets of layers, which can repeat. Thus with run-time parameter `--layer-order 1-24,8-32` Starling-LM-7B-alpha could be made to behave like Starling-LM-11B-alpha, without consuming any more memory than the 7B normally does. I also look forward to finding the limits of self-mixing. After it's done (which is taking way too long, mostly for stupid reasons) I'd like to start work on a llama.cpp plugin system for Guided Generation, which would work like grammars do now, but with arbitrary external logic instead of a grammar. After that, perhaps add a RLAIF feature to llama.cpp's finetune utility.