Depending on how much money you want to throw at the problem, all single CPU, in increasing cost: * Broadwell Xeon: E5-2699vr, E5-2698v4 or E5-2697v4 (I'd just go for the 99, the difference in price is negotiable. I'd say this is the oldest platform to consider because older ones don't support FMA nor F16c instructions, which are vital for CPU inference. You get four channels of DDR4-2400, which costs like 0.50/GB. There's no shortage of 2011-3 motherboards, but I'd stick with one that has IPMI and preferably ATX form factor to have an easier time fitting everything in a normal PC case. Aim for a board with 8 DIMMs so you can go for 512GB using 64GB DIMMs. This platform gives you 40 PCIe Gen 3 lanes. Most boards don't have an M.2 socket for NVMe, but that's fine. More on this in a bit.

• Cascade Lake Xeon: the refresh for socet LGA3647. much bigger CPU (physically). You get six channels of DDR4-2933 for almost double the memory bandwidth of Broadwell. Retail CPUs are quite more expensive than Broadwell, but there's a trick: Engineering Samples (ES), namely QQ89 (search on ebay). It costs about the same as E5-2699v4 but you get 24 cores. It has the same CPUID as retail models so BIOS compatibility is not an issue even with the latest BIOS. You also get AVX-512, but there's little evidence as of now if that speeds up inference on this platform. You get 48 Gen 3 PCIe lanes and a lot of boards expose two x16 slots. You also usually get at least one M.2 NVMe. Six channels give you 384GB RAM with 64GB modules. 2933 memory can sometimes be found ~0.70/GB, but 2666 can be found for ~0.60/GB without much hassle. It also supports Optane PDIMMs if you're into that (STH has a great guide on Optane PDIMM).

• Epyc Rome or Milan: this is the fastest DDR4 platform you can get without cost going up significantly. You get 8 channels of DDR4-3200 for 45% more bandwidth vs Cascade Lake. 3200 memory is at least 0.80/GB, but going for 2933 can lower your cost without much impact on speed. 2666 memory will further lower cost at a further reduction in speed. There are a few nuances with these CPUs: Epyc is for some reason allergic to Hynix LRDIMM memory with some boards. Samsung LRDIMMs are fine though. You can have 2-3 Hynix modules mixed with Samsung without issue, but YMMV. Epyc's real world memory bandwidth is heavily dependent on the number of CCDs in the CPU. You really need all 8 to be populated to get the maximum performance out of the memory controller. Easiest way to get that is to look for models with 256MB L3 cache. I'd say go for at least 32 cores. I have a bunch of 7642s (48 cores) and they're super nice and can be found cheaper than 32 core models (less known, I guess). For the extra cost you also get 128 lanes of PCIe Gen 4 to play with. ATX board models aren't as plentiful as the Xeons above because the CPU is even bigger than LGA3647 with two more memory channels, but those that are out there usually have at least three x16 slots. First Gen Epyc boards like Asrock's EPYCD8-2T or Supermicro's H11SSL will operate those slots at Gen 3 speed though. Beware of Dell/Lenovo locked CPUs. Unlocked CPUs become brand locked if installed on a Lenovo/Dell board. Also be careful of models that have a letter in the middle, some are heavily neutered, some don't work without a modded BIOS. These are Epyc issues only.

Which one of those to choose will depend on what you consider acceptable performance, even if you don't need real-time. This is a very subjective thing. I have all three in my home lab. I'd suggest looking for a good deal on a motherboard that ticks all the boxes you need, and building a system around that. A couple of months ago I got an Asrock LGA3647 board for 70 because it had some bent pins and a broken VGA output. The VGA is useless anyway since I use IPMI KVM anyways. The pins took 20 mins to fix using my phone's camera and a pair of fine tweezers.

For storage on any of these platforms, I strongly suggest skipping M.2 and going for HHHL PCIe NVMe storage cards (Google it). Those tend to be cheaper per TB, offer higher speeds, and at least an order of magnitude better endurance. Even one with 50% life left will have something like 10x endurance left vs a brand new consumer SSD of the same capacity. I have Samsung's 3.2TB PM1725 and they're great. HHHL cards plug into any PCIe slot and work as boot drives on most Broadwell boards (I'm not aware of any from a major brsnd that doesn't work).

Ebay has a lot of coolers for LGA3647, but if you want something quiet look for Asetek's LC570 LGA3647 on ebay. Been using them for over two years and really like them. For Epyc, socket SP3 is the same as threadripper TR4 and TRX40, so any cooler for any of those will work fine.

For GPUs, the best performance for the buck in the 24GB category is the 3090. They're down now to under 600 a pop if you're willing to search locally in classifieds. It also offers the easiest software setup. The best value for the money IMO is the Arc A770. It's down to around 200 for the 16GB model. Software setup is slightly more involved than the 3090 but not by much. Performance is great for the price. You can fit two 3090s or 3-4 A770s on a single ATX board depending on slot and lane organization. Arc really needs Rebar support to stretch it's legs. For that, ReBarUEFI is your friend. The project has a long motherboard compatibility list.

How much VRAM you'll need, and by extension how many GPUs, will depend on how many models you'll need to run in parallel (for ex: deepseek for text and Gemma 3 for image, maybe a 3rd for OCR). You'll also need to keep the KV cache at least at Q8 to have good performance, better yet fp16. That will impact how much VRAM you need. I'd suggest starting with one 3090 or two A770s. You can always add more if you feel you need the room.

For starters, download some models on whatever hardware you have and play with CPU inference using llama.cpp. Adjust the number of threads and throw some real use cases at it and watch the output speed. Don't pay attention to the response, just figure how slow is too slow for you, even if you don't need real-time. Also consider running multiple models for your different tasks, eg: Qwen 2.5 Coder or GLM-4 32B for coding, Gemma for image, etc. Me thinks you should spend quite a bit of time reading and understanding the software side of running LLMs locally before spending on hardware, to understand things like how model quantization and KV cache quantization, and model settings (temperature, samplers, etc) affect performance. Unsloth's doxumentation pages are your friend for figuring out the values of each model, and chatgpt is your friend to understand what each of those settings does.

Finally, keep in mind things are still changing very rapidly in this field. Everyone is moving to MoE models, smaller models are still getting increasingly smarter, and what might not be possible/true today could very much become reality in a couple of months.