How far would AMD Threadripper 3600 (24 core, 48 threads) and 256 GB of memory get me for running local LLMs?

Posted by x3derr8orig@reddit | LocalLLaMA | View on Reddit | 65 comments

I am thinking about buying this Black Friday a Threadripper to set up a local LLM inference machine. I would add some graphic(s) card, when the budget would allow me, but, this could be a start, no?

My reasoning is a "low" power consuption (280W), 256 GB of memory (enough to spare for some other tasks) and possibility to upgrade down the road. Without any discounts, this would be around 2200 euros (with cooling, case, a ton of disk space, etc, the whole package). I hope I could bring this down to at least 2000 or lower.

Does this make sense or am I delusional?