10k Hardware for LLM | TheaterFire

[-]

ttkciar@reddit

If you are new to the space, you really should start with models which work on the hardware you already have now, or with very modest upgrades (like a $50 8GB refurb GPU).

This will not only get your feet wet, but also give you a feel for what constraints are important for your use-cases, and learn your way around the software ecosystem.

Then, once you've gained some familiarity with the tech and gotten a good handle on what's important for your use-case(s), you can purchase the hardware most relevant to the constraints most limiting to you, from a position of some experience.

The hardware may well have become cheaper in the meantime, too.

[-]

Appropriate-Quit1714@reddit (OP)

What’s your go to sources for news and discussion regarding llm? You seem very knowledgeable

[-]

ttkciar@reddit

You flatter me. This very subreddit is pretty good (though it's been better). I've also seen some good hardware-related LLM discussion in r/HomeLab, theory discussed in r/MachineLearning and r/LearnMachineLearning, and some excellent training chat in r/Unsloth.

[-]

Appropriate-Quit1714@reddit (OP)

Thank you

[-]

Appropriate-Quit1714@reddit (OP)

Thank you for the reply. Actually just using an macbook air 2020, so I didn’t even bother to try with that lol But as I am considering to get a PC for some gaming too, I wanted to serve with the right hardware those two interests.

[-]

Long_comment_san@reddit

I wouldn't build anything nowadays. RAM and VRAM prices are through the roof. 10k will last you a lifetime on a good provider. If we go slightly back, I'd probably stack 4 RTX 3090ti and make some 8-12 ram platform. 96gb VRAM enough for anything realistic be it MOE or dense. That's probably dense 120B Q8 at blazing speed or something like deepseek easily.

[-]

Stepfunction@reddit

I'd get an RTX Pro 6000 and build around it. You'll be wanting 256GB of RAM, a server board with a server processor, etc.

[-]

Appropriate-Quit1714@reddit (OP)

Considering the price of 6000 now - wouldn’t make 2x 5090 make more sense?

[-]

QuantityGullible4092@reddit

No way in hell, always avoid multiple cards if you have the ability to. It massively slows things down and complicates the implementation

[-]

jettoblack@reddit

5090 prices have gone way up, and the RTX Pro 6000 price is stable or even gone down a bit.

Then there is power & cooling. 1 RTX Pro 6000 is easy to deal with. Running 3x5090s plus the host will require multiple 120V or special 240V dedicated circuits and PSUs if you’re in a 120V country like the USA, plus server-grade cooling (dedicated AC).

Not all ML tasks scale easily across multiple GPUs, or need special considerations. E.g. tensor parallel works on 2 or 4 GPUs but not on 3.

[-]

Arli_AI@reddit

Why? The Pro 6000 is cheaper than ever now.

[-]

QuantityGullible4092@reddit

This is the only answer, every other answer in this thread is nonsense

[-]

suicidaleggroll@reddit

Uh, have you seen prices lately? $10k can't buy you an RTX Pro 6000 and 256 GB of RAM, let alone everything else that's needed.

[-]

alphatrad@reddit

There goes your budget, lol. Imagine two of those bad boys!

[-]

oatmealcraving@reddit

I don't think GPUs run the fast Walsh Hadamard transform too well, or the optimal algorithm for GPUs has not been developed yet.

I would take that 10k and buy a multi-core server board (64 cores+)

Or possibly a cluster of imac mini's.

[-]

annon0976424@reddit

If power isn’t an issue grab 4x 5090s and a nice threadripper pro board with 128-256gb of ram

I open air run mine and you can do a ton with 128gb of vram and if you want

I built mine for about 10k with increased prices you can do it for about 11-12k

[-]

Appropriate-Quit1714@reddit (OP)

Thanks for the reply. Will have a look at that option. What kind of models do you run with that setup?

[-]

annon0976424@reddit

I switch around a lot depending on what I’m doing

Different models for different tasks.

The threadripper boards will give you 7 full pcie lanes. 4 of them are x16 pcie gen 4/5 depending on which one you get: Models can be loaded and unloaded super quickly which is great for my use.

I’ve trained with them for custom voice Lora’s and can run up to 70B models Int8 with room for good amount of context.

5090 is literally top dog for speed and has a solid amount of vram

[-]

annon0976424@reddit

You also don’t need a new threadripper

Can go for 5000 series

[-]

Ok-Internal9317@reddit

if I have 10k I'll spend 500 in openrouter for the next 5 years of my ai and then put the rest in nvidia stock

[-]

Correct-Gur-1871@reddit

I think much better would be to buy a single gpu desktop, rtx 5090 64gb 2tb gen 5 nvme ssd, for training models [less than 5000k]. and buy an m4 max 128gb or m3 ultra 96gb, whichever suits you [running llms 32b, 70b or 120billion gpt oss moe.].

[-]

Lachlan_AVDX@reddit

I'd wait for the m5 ultra 512gb and see if the specs are as good as expected.

[-]

alphatrad@reddit

Isn't there some bandwidth issue or something... I thought I saw someone review the Ultra 3 and while you could load huge models, the tps was like 8tps really low.

I could be misremembering and talking out my ass, so forgive me if I'm both wrong and dumb

[-]

Lachlan_AVDX@reddit

nah, you're good. I have an M3 ultra and the downside is that large contexts are really slow for TTFT. Depending on your application, this can be pretty limiting. With some of the top models, I've seen 5+ mins of TTFT for 60-100k contexts. This is because of the bandwidth being pretty low (300-400gb/s?)

There is some speculation that we may see 1000GB/s with the m5 ultra.

For me, the quality of the model is way more important than parsing long contexts. I can run GLM-4.6 on my ultra 3 studio at a 4 quant at 15-20 tps. I think for training and things like that, going NVIDIA makes a lot of sense, but the cost to produce anything great is just never going to be more worthwhile than renting GPUs. As a hobbiest though? I'm just so excited for the m5.

I want to see if they have a 1TB unified memory version, lol.

[-]

Appropriate-Quit1714@reddit (OP)

Would be interesting yes

[-]

Powerful-Street@reddit

This! I’ve heard some rumors of 1tb of ram and tensor core support, of some other name—in true Apple form.

[-]

DistanceSolar1449@reddit

Skill and performance increase isn’t linear to cost. You’re burning money above the $1-2k level.

Just buy 1-2 RTX 3090s. If you can’t make do with that, buy more. If you can’t fully utilize 1-2 3090s and train a model, you can’t do much with $10k anyways.

[-]

suicidaleggroll@reddit

What's your goal? Running small-medium models as fast as possible, or larger MoE models at an acceptable speed?

If it's the former, I'd get an RTX Pro 6000 and then the cheapest machine possible with at least 128 GB of RAM to drop it in. Should barely fit your budget, but as soon as you exceed the 96 GB of VRAM your speeds will drop like a rock from the shitty CPU.

If it's the latter, I'd drop back to more like 48 GB of VRAM with maybe 2x RTX 4090, and then go for something like an EPYC with at least 512 GB of fast DDR5. You'll run out of VRAM faster, but the CPU will actually be usable for inference on larger MoE models, making overall speed acceptable up to your full RAM capacity. Essentially, models <48 GB will be a little slower, models 48-120 GB or so will be a lot slower, and models 120GB+ will be faster when compared to the big GPU and shit CPU approach.

[-]

Jotschi@reddit

RTX pro 6000 Blackwell GPU.. 7.7k enough remaining for a decent CPU with memory. I would however not buy memory at the moment. Prices spike because the memory fabs slow production to buffer any looming ai bubble pop. They don't want to oversaturate the market I guess. (IMHO)

[-]

alphatrad@reddit

What kind of training are we talking about? You can do some basic training with $100 dollars worth of compute time.

https://github.com/karpathy/nanochat

But if I had 10k I'd probably build something with 2 or 4 5090's.

Then again for my needs since I'm willing to pay for SOTA models too. I'd probably just get 2 5090's and pocket the rest.

[-]

Appropriate-Quit1714@reddit (OP)

Thanks for your reply. Is the performance increase linear when using 1, 2 or 4x 5090?

[-]

alphatrad@reddit

you're increasing the vram which means loading bigger models

[-]

LilGardenEel@reddit

I would highly recommend you take some time to consider what it is you are trying to accomplish.

How large of a model do you want to run? How fast do you want the inference to be?

How much orchestration will be going on behind the scenes? Services, schedulers, data processing, caching, searching, indexing, etc.

I prioritized CPU for my needs and a single 4090 . Most I run on GPU would be a quantized 14B model

If you have eyes for inference on larger parameter models, def need additional/better gpu

thats my input~ I am not a professional; am hobbyist new to the space as well

[-]

jkh911208@reddit

M5 mac studio