Rant..
Posted by LingonberryMore960@reddit | LocalLLaMA | View on Reddit | 17 comments
I recently bought an awesome machine and I’m now able to run larger models. Since I’m new to all of this, I did my research and realized that the whole local AI scene is an absolute mess. Suddenly, a shit-ton of stuff needs to be installed on my computer, and it’s impossible to keep track of where everything went. To be fair, some things give you at least a bit of control, but because of all "dependencies" aka bloatware, I ended up having to reinstall Windows.
Is it really possible that everything around AI is this clunky and annoying? Can’t they just make a simple piece of software with plugins if you want something more advanced than just chatting? This maze of nonsense is disgusting.
BeepBeeepBeep@reddit
Use llama file
Tartarus116@reddit
Reminds me of the guy who complained about there not being an exe lol https://programmerhumor.io/git-memes/i-dont-give-a-fuck-about-the-fucking-code-nvqn
LingonberryMore960@reddit (OP)
Somehow explains my frustrations hahaha
Tartarus116@reddit
Just dockerize everything ffs
lisploli@reddit
Well of course, there is an easy solution. Buy some online service. But if you do not buy a product, you do not get professional polishing.
There are a few nice packages, depending on what you want to do. But many of the tools are very small and tailored to solve specific problems, because they are made by individuals in their free time. They do not have the intention to provide disgusting shit or bloatware, but they are limited and thus have to depend on other, similar tools to get things done. This has worked well for decades and will continue to do so.
Consider learning a bit of the python workflow. Not coding, but deployment techniques. Also, linux allows for a much cleaner experience e.g. failed experiments won't affect the system. That's not necessary, but comfy.
Awwtifishal@reddit
llama.cpp and programs based on llama.cpp are self-contained and don't require python stuff etc. so try something like jan.ai which include llama.cpp, a GUI, a way to download models from within, etc.
Saber-tooth-tiger@reddit
I rarely install stuff, I install docker and run everything as services using docker. This way your computer stays clean and can run things without conflicts with other software on it. There are other benefits to using docker, like using docker compose to running multiple dependent services with certain configs with just one command.
cromagnone@reddit
1) Second hard drive, install linux.
2) Get your head around uv and managing virtual environments.
You're right, the whole ecosystem is a mess. But it's clearer in native linux as WSL is, in my experience, an extra bit of mess to deal with.
shaakz@reddit
i was gonna write exactly this. Dualboot linux and use uv to manage your python envs, clean and organized. Plus its a skill worth learning
sxales@reddit
That is not AI, that is just how python works. Use virtual environments.
Or use llama.cpp, koboldcpp, LM Studio, Jan, any other program with pre-compiled binaries.
No-Mountain3817@reddit
I assume you are not familiar with
venv.It’s not the most powerful option, but it’s one of the simplest ways to manage virtual environments.
Just run
workon VENV_NAMEfrom anywhere before installing packages, and you’ll keep everything clean and organized.
https://virtualenvwrapper.readthedocs.io/en/latest/index.html
thesuperbob@reddit
Yeah it's a bit of a wild west situation in terms of software and hardware. Windows isn't great for running anything unusual, some interesting GPUs don't even have Windows drivers.
That said, it's as much on you for messing up your system as it is on people spreading unreasonable hype and making this seem easy even though it isn't. Look past those hyper-optimistic posts and there's plenty of things that don't work, or barely work and you don't see all the crashes posted next to someone's highlight reel.
Like others said, look into isolating your AI sandbox somehow, or just get a second SSD and install Linux. Some distributions already come with a package for things like llama.cpp that automatically install all the required dependencies, stuff should work out of the box on a fresh install.
So yeah, take it easy and give yourself a moment to figure out how to get the most out of the hardware you got. It gets better once you get something up and running. At least until you mess with it after some hot new model drops, than it's more trial and error and looking for people posting their settings and tricks. But the upside is that it's all yours, $/token, no rate limiting, nobody gathering analytics on what you use the model for.
dsanft@reddit
You can run llama-cpp in a Docker container. That takes care of like 95% of the annoyance.
swagonflyyyy@reddit
No, not everything is clunky and annoying. I think you're doing it wrong.
1 - Get a second hard drive. 2 - Use env variables as needed to point the model downloads to a folder in that second hard drive. 3 - For a simple experience, get LM Studio, its literally plug and play. You'll be able to download models there. 4 - If you're messing with python scripts, learn how to create a virtual environment to pip install all the dependencies you need. Dependency hell is a python problem, not an AI problem.
This is a simple way to get started.
bigattichouse@reddit
If you're playing directly with python, make sure you use "virtual environments", I didn't know about them until recently and it's absolutely simplified my life - instead of changing the machine's allowable packages, you can create
RiotNrrd2001@reddit
Install LM Studio. Then, inside LM Studio, look for the models you want and download them.
That one thing to install, plus any LLMs (which are installed from inside the application). Not complicated at all.
bucolucas@reddit
Use AI to help set it up :)