Built my first AI + Video processing Workstation - 3x 4090
Posted by Special-Wolverine@reddit | LocalLLaMA | View on Reddit | 221 comments
Threadripper 3960X ROG Zenith II Extreme Alpha 2x Suprim Liquid X 4090 1x 4090 founders edition 128GB DDR4 @ 3600 1600W PSU GPUs power limited to 300W NZXT H9 flow
Can't close the case though!
Built for running Llama 3.2 70B + 30K-40K word prompt input of highly sensitive material that can't touch the Internet. Runs about 10 T/s with all that input, but really excels at burning through all that prompt eval wicked fast. Ollama + AnythingLLM
Also for video upscaling and AI enhancement in Topaz Video AI
irvine_k@reddit
Is there a LLaMa 3.2 70B?
Special-Wolverine@reddit (OP)
Not yet. 1B text, 3B text. 11B vision, and 90B vision for now
irvine_k@reddit
It's just that I saw you mention it like that, so I got excited.
Special-Wolverine@reddit (OP)
Oops. Just noticed my typo
auziFolf@reddit
Beautiful. I have a 4090 but that build is def a dream of mine.
So this might be a dumb question but how do you utilize multiple GPUs? I thought if you had 2 or more GPUs you'd still be limited to the max vram of 1 card.
IT PISSES ME OFF how stingy nvidia is with vram when they could easily make a consumer AI gpu with 96GB of vram for under 1000 USD. And this is the low end. I'm starting to get legit mad.
Rumors are the 5090 only has 36GB. 36GB.... we should have had this 5 years ago.
Special-Wolverine@reddit (OP)
In probably 2 years there will be consumer hardware that has 80gb VRAM but low TFLOPS made just for local inference, until then you overpay.
As far as making use of multiple gpus, Ollama and ExLlamaV2 (and others I'm sure) automatically split amongst all available Gpus if the model doesn't fit in one card's vram
Themash360@reddit
I’m honestly surprised there are no high vram low compute cards from nvidia yet. I’m assuming it has more to do with product segmentation than anything else.
Shoddy-Tutor9563@reddit
Chinese brothers have modded 2080 and put 22 Gb of vram there. Google it
claythearc@reddit
Maybe - inference workloads are pretty popular though and don’t necessarily need anything proprietary* (some do w/ flash attention) so if it were something reasonably obtainable to make amd/intel would release one, I would think
BhaiMadadKarde@reddit
The new Macs are probably filing this niche right?
Special-Wolverine@reddit (OP)
Their inference speed is on par, but prompt eval speed burning through 40K word prompts is about 1/10th the speed
chrislaw@reddit
I'm really curious what it is you're working on. I get that it's super sensitive so you probably can't give away anything, but on the offchance you can somehow obliquely describe what it is you're doing you'd be satisfying my curiosity. Me, a random guy on the internet!! Just think? Huh? I'd probably say wow and everything. Alternatively come up with a really confusing lie that just makes me even more curious, if you hate me, which - fair
Special-Wolverine@reddit (OP)
Let's just say it's medical history data and that's not too far off
chrislaw@reddit
Oh cool. Will you ever report on the results/process down the line? Got to be some pioneering stuff you’re doing. Thanks for answering anyway!
SniperDuty@reddit
Does CUDA work ok?
babeal@reddit
Tensor parallelism. Splitting layers or attention heads across cards.
auziFolf@reddit
Thanks, I'm currently doing a deep dive into this.
Obvious-River-100@reddit
It would be cool if they made a card with a 4090 GPU, eight DDR5 slots, and no HDMI or DP ports . In principle, such a card would cost around $1000.
kkchangisin@reddit
It would be extremely slow. The fastest DDR5 I could find from a quick Google is this PoC:
https://www.techradar.com/computing/computing-components/gskill-shows-off-fastest-ever-ddr5-ram-that-hits-incredible-speeds-at-computex-2024
10600 MT/s is 84.8 GB/s per channel.
RTX 4090 is 1008 GB/s (3090 is still 936 GB/s). You'd need 12 channels of the fastest DDR5 on the planet that you can't even buy to reach that.
If Nvidia completely lost their minds and offered such a bizarre thing they'd sell so few of them (a few thousand?) they would either be an extreme loss-leader or cost many multiples of $1k.
Obvious-River-100@reddit
I suggest you have 50x4090 GPUs at home, and you can easily run a 405B FP16 model, while I would be fine with this card and 1TB of DDR5 memory for that.
kkchangisin@reddit
Fortunately Intel is doing quite a bit of work with "AI instructions", die space for dedicated AI, etc on CPU - that's going to be the only way you're going to use socketed memory (just like today but faster).
I try to be realistic ;).
kakarot091@reddit
We feel you bro. That's why monopolies are bad.
MoffKalast@reddit
Monopolies are bad, but AMD existing just to keep antitrust away from Nvidia so they can fully utilize their monopoly with impunity is even worse.
kakarot091@reddit
Well, Lisa and Jensen are cousins after all lmao.
MoffKalast@reddit
I would be really surprised if they didn't decide to segment the market over a family dinner lol. Nvidia takes the GPU sector and leaves the CPU turf alone, AMD mainlines CPUs and does only the bare minimum for Radeon. Win-win.
kakarot091@reddit
I doubt it's that way but definitely funny to think about.
kkchangisin@reddit
I feel you but we just need to accept this is a complete pipe dream whether it's Nvidia, AMD, Intel, or anyone else.
Not only is this price point not anywhere close to reality already you need to factor in what it takes to get a product to market:
Every single model/SKU has costs. The only way it works is to make it up in volume...
Nvidia ships something like 8 million desktop GPUs per quarter (32 million/yr). This sub has 225k subs - even if everyone here bought such a card within the first year that would be an increase of 0.7% additional units shipped and needless to say it would tear into higher margin datacenter GPU products.
Nvidia gets a lot of hate but the fact is they're no more "greedy" than any other large publicly traded company. They're selling products at a price point the market will bear and their 80% market share on desktop and > 90% market share in datacenter reflects that. Literally any company would be doing exactly what Nvidia is if they could.
The reality is our use case is a tiny, tiny, tiny niche (irrelevant) market when you zoom out. Until other use cases from the broader and much larger market call for more VRAM this will never happen. They sell GPUs with 80GB for tens of thousands of dollars at 100% gross margin and still can't keep up with demand.
In the current climate something this dumb and outrageous (from the perspective of Nvidia) would be a "Jensen gets fired by the board" level blunder.
Don't hate the player, hate the game.
NoAvailableAlias@reddit
32 is the rumor, would mean the RTX A6000 BW "should could" be 64gb at over 9000 monies knowing ngreedia... sad because RDNA4 won't have near the memory bandwidth to hold any candle even if you can buy eight 16gb cards for a mining mobo for the same price...
man_eating_chicken@reddit
Noob here. Just lurking until I can afford a machine that can handle LLMs.
What are the pros and cons of running 3 4090s with power limits over 2 without?
Special-Wolverine@reddit (OP)
All that matters for large LLM models is absolute amount of VRAM. I could probably achieve the exact same results with 4x cheaper 16Gb GPUs considering my needs are about 64Gb to run Llama 3.1 70B 4bit + max context window, but then wiring and cooling 4 16Gb cards would probably be harder than 3
man_eating_chicken@reddit
Thanks!
Lutr4phobi4@reddit
I wonder what effect PCIe lanes have here? Adding NVMe and filling these many lanes can choke bandwith in these types of builds. Thoughts from anyone that want's to dive in here?
Special-Wolverine@reddit (OP)
A second 4090 on a z690 motherboard with i9-13900K was not happy running at x4 through the chipset while the first one was at x16 from the CPU. That's why I changed to the threadripper with 64 PCIe lanes
Lutr4phobi4@reddit
Thank you for that reply. I have a Rog Strix Z790-e gaming board and I am struggling to balance the PCIe lanes with my 4090 and NVMe drives 1 gen 5 (which shares the same lane as the first PCI slot). It balances through bifrucation. But the 4090 is a gen 4 so on a gen 5 x8 is essentially x16 on a gen 4 board while the nvme really only needs x4 channel. Plus I have all the other lanes populated with Gen 4 NVMes. It's a nightmare. I may consider moving over to AMD due to this. Plus the memory controller on Z790 can't run dual channel memory at full XMP profile for DDR5. Infuriating.
Special-Wolverine@reddit (OP)
Yes, if you want max performance out of multiple NVMEs plus multiple GPUs, you gotta leave consumer level Intel stuff. Either server Intel stuff or AMD Threadripper or EPYC.
Lutr4phobi4@reddit
Yea. It sure is looking this more and more. UGH. Peace!
Special-Wolverine@reddit (OP)
I actually decided against the 3 GPUs because of diminishing returns.
Kooky-Height-7382@reddit
DIY case, 50€, will fit an elaphant and you can dry your hair topside
Special-Wolverine@reddit (OP)
Noice
eyeseesharp@reddit
How does this compare performance wise with ChatGPT 4o for example?
Special-Wolverine@reddit (OP)
Use Groq or Venice to try out the open source LLM models for output content quality if that's the kind of performance you are talking about. The speed in tokens per second of 4o is constantly improving, so that's hard to answer if that kind of performance is actually what you're asking
Mysterious-Name-6304@reddit
This may seem like a dumb question, but if I build a kick ass AI image rendering rig, does that mean it will automatically be a kick ass gaming rig, too?
Special-Wolverine@reddit (OP)
Gaming can only benefit from one GPU these days
julien_c@reddit
Very nice setup!
Armym@reddit
Clean for a 3x build
Special-Wolverine@reddit (OP)
Wanna replace all the 12VHPWR cables with 90 degree CableMob ones for much less of a rat's nest and maybe a chance of closing the glass if the Suprim water tubes can handle the bend
Armym@reddit
I saw that you are not impressed with the tokens per second. Try running vLLM and see if it gets better. Also, look for the George Hotz RTX 4090 p2p driver. It boosts inference quite a lot.
SniperDuty@reddit
Legend, never knew about this. This would not be beneficial for 4090 Suprim X which I'm thinking of getting though would it?
Armym@reddit
Why wouldn't it be beneficial? If you have multiple 4090s, it enables basically nvlink without nvlink. (Runs through PCIe)
SniperDuty@reddit
Updated my comment, it only does this via native Linux though not Windows WSL is my understanding
Armym@reddit
Yeah. I bought a second nvme drive and dual boot ubuntu.
bbsss@reddit
Did not know he published it. Thanks a bunch!
Special-Wolverine@reddit (OP)
Thanks, will definitely look into it. Only just got this finished and now I'm going to try all the different front ends and back ends that support GPU splitting
philmarcracken@reddit
at this point just buy that 3M Novec fluid lol, you've spent so much already.
antialtinian@reddit
Are you trying to set your new rig on fire!?
CableMod_Alex@reddit
If anything, our cables will make that build look cleaner and reduce the potential points of failure thanks to direct connections. I think you may be confusing our cables with our angled adapters which have been recalled nearly a year ago. :)
horse1066@reddit
and as if by magic, a CableMod fairy appears...
TechOverwrite@reddit
I hear that if you say CableMod three times on a full moon, they send you a 4090 for free.
CableMod_Matt@reddit
Got lost in the mail, sorry. Darn FedEx!
EDLLT@reddit
I'd highly recommend using langflow instead of AnythingLLM
Special-Wolverine@reddit (OP)
Thanks, I'll try it out. That's the crazy thing about this time we live in - everything is still up for grabs. The best solution to any given problem is very likely unknown by the people trying to solve that problem.
Armym@reddit
If you made a document/book about best llm practices, you would have to update it every half a month or so.
satireplusplus@reddit
For a moment I was confused because I was seeing 4 GPUs.... but looks like its just three fans at the top.
horse1066@reddit
I read it as 4 x 4090 and spent 30 seconds playing Where's Wally :/
Master-Pizza-9234@reddit
Can you show a diagram of the radiator positions? Since it seems like you have 3 liquid cooled components but can only place a rad safely on the side intake and top exhaust. Hopefully not a rad mounted at the bottom, remember that the air inside the loop rises, so having a rad below is almost always a bad idea for cooling since it equals air where the heatsinking is supposed to happen
Special-Wolverine@reddit (OP)
Didn't know this and has been pointed out in replies, so I'm very grateful and will change it
bbsss@reddit
Connected my 3rd 4090 yesterday. The speed went down for me on my inference engine (vLLM). It went from 35t/s to 20t/s on vLLM on the same 72b 4bit. That's because odd number gpu's can't use tensor parallel if the layout of the llm doesn't support it, so then only pipeline parallel works. However it did become a LOT more stable for many concurrent requests, which would frequently crash vLLM with just two 4090.
Hooking up a 4th 4090 this week I think, I want that tensor parallel back, and a bigger context window!
smflx@reddit
Tensor parallel is of 2, 4, 8 gpus. Not just even number as i understand. Precisely, # of attention heads should be divisible by # of gpus.
bbsss@reddit
Thank you, that is an important distinction I wasn't sure off. Now I won't make the mistake of buying two more 4090 to push it to 6.
Special-Wolverine@reddit (OP)
Ooh, interesting. I thought the tensor parallelism only mattered for training
RipKip@reddit
Haha great excuse to buy another
Kooky-Height-7382@reddit
This will fit another 4090....
_KingDreyer@reddit
may i ask the subject matter of this sensitive material or is that confidential too?
Sad-Objective-8771@reddit
Can you share build cost?
MoffKalast@reddit
I doubt OP wants to look at their wallet for a while after this. Gotta let it recover a bit first.
alpetera@reddit
Good looking rig! How's the elec bill for that by the way?
PoliteCanadian@reddit
For that money you could have bought an MI300X machine that would be about 5x as fast and have waay more (and faster) vram.
SpinCharm@reddit
No. They cost $15000. Unless you’re Microsoft and other companies buying them in bulk in which case they’re $10000.
Special-Wolverine@reddit (OP)
$6,500? 15x?
Wrong-Barracuda0U812@reddit
Are you using this rig to smooth out gimbal shots or to upscale old/new footage? I’m new to this space only use Foocus locally to train txt to img on a Asus 4070tiS, small in comparison to this beast.
Special-Wolverine@reddit (OP)
Upscale old home movies as one use case. The other video processing use case would give away my profession, which I'd rather not
Wrong-Barracuda0U812@reddit
No worries I used to work for ProApps at Apple and then on Davinci as a hardware SQA, most of my life as hardware SQA something. I’m still not clear why it takes so much processing power to essentially transcode video in AI but I’m beginning to learn.
SniperDuty@reddit
OP get the Corsair Premium 600W PCIe 5.0 GPU power connector then you can close the case. Also what case is that?
Special-Wolverine@reddit (OP)
I had two of the Corsair 12VHPWR cables when it was just two GPUs and a 1000W Corsair PSU. Will get 12VHPWR cables for my 1600W EVGA. Case is NZXT H9 Flow, but gonna change to Lian Li o11 dynamic Evo XL with front mesh kit. 900mm PCIe riser routed behind the mobo.
Anmolsharma999@reddit
Are you in a dev job?
Special-Wolverine@reddit (OP)
No, government work
CheatCodesOfLife@reddit
You'd get like 30 with exllamav2 + tp
Special-Wolverine@reddit (OP)
That's definitely the next step . But I was getting errors installing ExLlamaV2 for some reason
noneabove1182@reddit
are you on linux?
I've had good success with exl2/tabby in docker for what it's worth
Special-Wolverine@reddit (OP)
No, Windows. Kind of a noob to this with zero coding skills, so Linux is intimidating
idnvotewaifucontent@reddit
MX Linux (KDE Plasma version) has a very Windows-like experience. It's the one I've stuck with more or less permanently as a daily driver after trying Ubuntu, Cachy, Zorin, Pop, and Mint.
The terminal app in MX allows you to save commands and run them automatically so you don't actually need to remember what syntax and commands do what.
Special-Wolverine@reddit (OP)
Interesting. Will definitely look into this
genshiryoku@reddit
Just so you know Linux is extremely approachable for someone without coding skills. If you have the technical know-how to host local models and build PCs then you can handle Linux just fine.
I recommend a rolling distro like Arch. Because you're a noob I would recommend EndeavourOS.
The funniest thing you will experience is that Linux will most likely feel easier to use and more convenient to Windows after just 1 month of using it.
noneabove1182@reddit
Ah fair, you should definitely consider it, it's not as bad if you use it as a server and not a daily driver, but only if you feel like experimenting :)
Special-Wolverine@reddit (OP)
Yeah, need it for a lot of other things like Whisper AI transcription, ThinkOrSwim stock charting, Google web messages, etc...
Nrgte@reddit
Install Ooba, it comes with Exllama and TP. Although I haven't found a way to increase performance with TP. Not sure how it's supposed to work.
princetrunks@reddit
Amazing. My build \~10 years ago was about $3000 for my AR/VR work and was 2 1080s. Was almost the power of a PS5 is now but this is the kind of next upgrade I'd love to do now for my job/business.
ofmoneygrab@reddit
Whats the cost of this build?
Special-Wolverine@reddit (OP)
$6,500. The breakdown is elsewhere in the replies
EmilyAnderson172@reddit
What kind of main-board you have used?
Special-Wolverine@reddit (OP)
Asus ROG Zenith II Extreme Alpha sTRX4
nanomax55@reddit
Are you considering a bigger case ?
Special-Wolverine@reddit (OP)
Absolutely. Lian Li o11 dynamic Evo XL with front mesh kit
BakerAmbitious7880@reddit
If you are using Windows, check your CUDA utilization while running inference, then probably switch to Linux. I found on a dual 3090 system (even with NVLink configured properly), that when running on two GPUs, it didn't go faster because CUDA cores were at 50% on each GPU, while I was getting 100% when running in one GPU (for inference with Mistral). Windows sees those GPUs as primarily graphics assets and does not do a good job of fully utilizing them when you do other things. The hot and fast packages and accelerators seem to be only built for Linux. Also, if you haven't already, look into the Nvidia tools for translating the model to use all those sweet sweet Tensor/RT cores.
Special-Wolverine@reddit (OP)
Great tips. Will look into that stuff
kkchangisin@reddit
FYI in terms of TensorRT on my 4090s I see roughly 10-20% performance improvement over vLLM. You've mentioned making it available via network so you'll probably end up with Triton Inference Server + TensorRT-LLM but be aware - it's a BEAST to deal with to the point where Nvidia offers NIM so mortals can actually use it.
If you absolutely need the best perf or are running hundreds of GPUs the level of effort is worth it (better perf = fewer GPUs for the same volume of traffic). Otherwise just save yourself a ton of hassle and use vLLM - they're doing such great work over there the 10-20% gap is closing on the regular.
Special-Wolverine@reddit (OP)
Good tips. Thanks
SniperDuty@reddit
How do you check CUDA utilisation? Code it alongside a run?
BakerAmbitious7880@reddit
There are some more advanced Nvidia tools that you can use (Nsight) to get really robust data, but you can also get rough values from Windows Task Manager (Performance Tab, Select GPU, Change on of the charts to CUDA using the dropdown). This screen shot is running inference on a single GPU, but it's not quite to 100% because it's running inside of a Docker container under windows.
horse1066@reddit
I hadn't actually realised that you could swap one for a CUDA graph, thanks for the tip
ArtyfacialIntelagent@reddit
Great tip, I never noticed the dropdowns! But this is what I see on Win11. Is any of these CUDA in disguise?
kkchangisin@reddit
NICE!
You basically built your own Lambda Labs Vector workstation - down to the MSI Suprim. Then wedged in a 4090 FE for good measure :).
If I shipped you my Vector do you think you could get a 4090 FE in there for me ;)?
Special-Wolverine@reddit (OP)
Ha, never even seen that one but you are right. Almost the exact same hardware. The 3rd card has entirely diminishing returns on performance besides simply making it possible to run 70B at max context
satireplusplus@reddit
Whats the T/s in llama.cpp ? Also not sure if you are aware of it, but you can run many independent concurrent sessions before you saturate compute on the GPUs. As all things LLM, memory speed is nearly always the bottleneck, see https://www.theregister.com/2024/08/23/3090_ai_benchmark/
Special-Wolverine@reddit (OP)
Haven't used llama.cpp yet - next step is to test all the front and back ends
satireplusplus@reddit
Did you use a Pci-E cable extender for the third card?
Special-Wolverine@reddit (OP)
Yup. 900mm routed behind the mobo
nero10579@reddit
I really don't think it's a good idea to leave the pcie plugs unplugged on 4090s.
Special-Wolverine@reddit (OP)
Multiple sources say 3 of the 4 is fine
nero10579@reddit
Yea and I thought 4 out of 4 is fine until my 4090 burned. I now use a real proper 12-pin cable.
randomanoni@reddit
Oh shit your 4090 burned? Did you power limit? I don't see many horror stories like that in here. It might be worth it to make a separate post about "LLM gone wrong".
nero10579@reddit
No I maxed the power limit like I do with all my GPUs. I expect it to be able to do that.
To be fair if you just use your gpu for inference it’s probably fine. I was training models on it for days on end and I probably should have upped the fan speed a bit.
Special-Wolverine@reddit (OP)
I'm going to be ordering custom 90 degree 12VHPWR cables from CableMod
nero10579@reddit
You can just get then from amazon. I got them from there.
vrweensy@reddit
which models do you use most locally?
Special-Wolverine@reddit (OP)
Llama 3.1 70B Instruct is best for the type of prompts I do for work, but Claude 3.5 sonnet is best for non-sensitive material
Al-Horesmi@reddit
How did you mount the third card?
Special-Wolverine@reddit (OP)
There's a slot in the bottom of the case which the protruding portion of the card's bracket sticks through. I then secured it in place with bolts and nuts to keep it from being pulled back up through that slot. Then there's a 900mm PCIe riser that runs behind the mobo to the GPU
Proud-Discussion7497@reddit
How much did you pay for all?
Special-Wolverine@reddit (OP)
$6,500. Breakdown is elsewhere in replies
LuciiFlynn@reddit
You're not serious!
This is your first built?
LIke ever?
I'm soooo jelly!
I only have a rtx 4070 😓
Special-Wolverine@reddit (OP)
First AI rig build. Only ever built two budget home theater PC's before. with all the time savings I get out of AI, I have a lot of spare time to tinker
IloveMarcusAurelius@reddit
What time savings do you get from AI?
Special-Wolverine@reddit (OP)
No exaggeration - projects that used to take me 8 hours now take 3 minutes + maybe 15 minutes of final editing
IloveMarcusAurelius@reddit
How did you learn AI? I want to get into it. Seems like you know your stuff
Special-Wolverine@reddit (OP)
First step is to read Anthropic's best practices documentation on their website - that's where I started and learned that your prompt should always have sections for purpose, tone, examples, and input content - all separated by XML tags. Subscribe to Claude or ChatGPT and start building Projects (Claude) or GPTs (OpenAI) to help with all the little tasks you do at work and home. For example, last night I was applying to private kindergartens for my daughter. I pasted all of that school's mission and values from their website into one document, then added a section with all the essay answers I wrote about my older daughter to teach it my writing style , then gave the LLM all the essay questions for the new school application and told it to answer them in my writing style incorporating the mission and values of the school, and to give me five options for each question. I could then pick and choose the answers that were closest to my younger daughter's personality.
Ginkgopsida@reddit
This is so awesome. How did you connect the third PCIe slot?
Special-Wolverine@reddit (OP)
900mm PCIe riser from the bottom slot around behind the mobo to the vertical GPU
Ginkgopsida@reddit
That's awesome. Thanks. Is there a way for them to share the VRAM?
FarFun1@reddit
Is that for commercial, professional reasons or just personal/hobbyist stuff?
Special-Wolverine@reddit (OP)
Legal and professional reasons
Silent-Wolverine-421@reddit
Good one. Glad someone used threadripper. I hope you got to make all three GPUs work in x16 mode?
Right?
Special-Wolverine@reddit (OP)
Only two of them. Third in x8 😞
Silent-Wolverine-421@reddit
My wolverine bro !! Check cpu lanes on your threadripper. I think you should be able to run all on x16. Check once please.
Special-Wolverine@reddit (OP)
The 3960X has enough lanes, but the Asus ROG Zenith II Extreme Alpha motherboard can only do x16 - x8 - x16 - x8
_lMark@reddit
So he has a threadripper, not a i9 13900k?
Silent-Wolverine-421@reddit
Thats what he has written under the image
_lMark@reddit
Are you sure it runs in x8? I looked up, i9 13900k has only 16 CPU lanes, having 3 GPUs means you have to cover 48 PCIE lanes, you can't do that, that's why you have to reduce it to probably something like 4x at max. Every GPU gets only around 4 CPU lanes (you can only go 4x, 8x, 16x), not 8x... That's why CPUs like Threadripper exist, because they got a lot of CPU lanes and can get a lot of bandwidth connection with PCIE devices.
joeen10@reddit
What do you use it for? I'm curious
intheshad0wz@reddit
Jesus Christ 🤤
hamada147@reddit
That is very cool 😎
I would love to upgrade my setup to that but I’m honestly waiting to save up and for the 5090 graphic card to be worth it as it will be 32 vram (finger crossed) each and with 3 of them it will be epic 🤗
I would also use a different motherboard ASUS workstation and fill it with 1 tb ram
Of course I’m gonna start small and move my way to that specifications
Mithgroth@reddit
I suppose you preferred 4090s over A40 or A100 for video processing, right?
Abhrant_@reddit
absolutely not GPU poor
Parking_Bluebird826@reddit
How many gb's of vram?
Special-Wolverine@reddit (OP)
72
Kooky-Height-7382@reddit
Your electricity bill must be brutal....
Special-Wolverine@reddit (OP)
Under my desk at the office...
tommitytom_@reddit
Where is the PSU? ;)
Additionally, did you find multiple GPU's sped up inference in Topaz? I was surprised how slow it was on a single 4090 and wasn't using anywhere near it's full capacity (according to power draw)
Special-Wolverine@reddit (OP)
PSU is in a second chamber behind the mobo.
Topaz is not sped up unfortunately. Probably the biggest disappointment. Might have to find a video upscaling and enhancing software that better takes advantage of GPU scaling
Javunavonx@reddit
Need 😝
Darkonimus@reddit
Wow, that's an absolute beast of a build! Those 3x 4090s must tear through anything you throw at them, especially with Llama 3.2 and all that video upscaling in Topaz. The power draw and thermals must be insane, no wonder you can’t close the case.
Special-Wolverine@reddit (OP)
Honestly a little disappointed at the T/s, but I think the dated CPU+mobo that is orchestrating the three cards is slowing it down, because when I had two 4090s in a modern 13900k + z690 motherboard (the second GPU was only at X4) I got about the same tokens per second, but without the monster context input.
And yes, it's definitely a leg warmer. But inference barely uses much of the power, the video processing does though
jrherita@reddit
Were the two GPUs running at full power? 3 x 300W cards vs 2 x 450W might not show much difference.
Special-Wolverine@reddit (OP)
Power limiting GPUs has no effect on inference because unrestrained they only pull about 125W each during inference
Some_Endian_FP17@reddit
What's your GPU utilization during inference? 125W each sounds like 50% utilization for each GPU, so LLMs are more memory-constrained than compute-constrained.
Special-Wolverine@reddit (OP)
GPU utilization in task manager is like 3% during inference with a spike to like 80% during the 30 seconds or so of prompt eval
Some_Endian_FP17@reddit
Holy crap. So prompt eval depends on compute while inference itself is more about memory size and memory bandwidth.
This market is just asking for someone to come up with LLM inference accelerator cards that have lots of fast RAM and an efficient processor.
Special-Wolverine@reddit (OP)
💯💯💯
jrherita@reddit
Interesting - I've only just started getting into this and noticed LLMs were very spikey on my 4090.
Is it possible you need more PCIe bandwidth per card to see better scaling with more cards?
randomanoni@reddit
Try TP. Sweet spot is around 230W for 3090s at least, not sure what changes with 4090s.
PoliteCanadian@reddit
Because they're 4090s and you're bottlenecked on shitty GDDR memory bandwidth.
All the datacenter AI devices have HBM for a reason.
clckwrks@reddit
That’s because it is not utilising all 3 cards. It’s probably just using 1.
I say this because of NVlink not being available for 4 cards
Special-Wolverine@reddit (OP)
No, it runs about 21Gb of VRAM on each card for 70B. The large context is what's slowing it down
aaronr_90@reddit
I would be willing to bet that this thing is a beast at batching. Even my 3090 gets me 60 t/s on vllm but with batching I can process 30 requests at once on parallel averaging out to 1200 t/s total.
Special-Wolverine@reddit (OP)
Gonna run LAN server for my small office
NoAvailableAlias@reddit
Increasing your model and context sizes to keep up with your increases in vram will generally only get you better results at the same performance. All comes down to memory bandwidth, future models and hardware are going to be insane. Kind of worried how fast it's requiring new hardware
Special-Wolverine@reddit (OP)
Understood. Basically for my very specific use cases with complicated long prompts in which detailed instructions need to be followed throughout large context input, I found that only models of 70b or larger could even accomplish this task. Bottom line was that as long as it's usable, which 10 tokens per second is, all I cared was about getting enough vram and not waiting 10 minutes for prompt eval like I would have with the Mac Studio on M2 ultra or MacBook Pro M3 Max. With all the context, I'm running about 64gb of VRAM.
HelpRespawnedAsDee@reddit
Or how expensive said hardware is. I don’t think we are going to democratize very large models anytime soon
NoAvailableAlias@reddit
Guarantee they won't just sunset old installations either... Hek now I'm worried we don't have fusion yet
t0lo_@reddit
lol nice, i caught you
Darkonimus@reddit
I wish, lol.
dhrumil-@reddit
Specs please 🥺
Nickbot606@reddit
Do your lights dim slightly every time that thing turns on? Wouldn’t it cost less at that point to just hire an assistant? 😝
Whispering-Depths@reddit
I would have just gone with an A100 80GB at the cost of making this rig lol, they are $7k-11k tops.
United-Advisor-5910@reddit
If the masterrace was a race you would be the winner.
Themash360@reddit
You madman. How is the 3rd gpu connected ?
AmphibianHungry2466@reddit
Amazing! What OS you run?
Special-Wolverine@reddit (OP)
Windows. Would run much faster in Linux from what I've read
GeminiDroidAtWork@reddit
Wow, super cool!!! Congratulations on the setup. Do you plan to write a blog on how you did the whole setup from scratch, along with the overall cost? It will help newbies like me, who are planning to do their own setup at some point.
Special-Wolverine@reddit (OP)
I should, but alas I wasted far too much time building it, and now I have to get back to work!
But I have actually explained a lot of it here in replies if you look around
Beastdrol@reddit
So jelly that’s a super nice build.
Lots of compute power too for ai inferencing.
Have you tried fine tuning any models out there; what sort of performance did you get?
Special-Wolverine@reddit (OP)
Thanks, Just finished it. Fine tuning definitely in the plans
_lMark@reddit
Hello Special Wolverine, What kind of CPU are you using?
aniketmaurya@reddit
Dope
maximthemaster@reddit
beautiful have fun. vhpvr cables are so sensitive nice to see you made it work.
cpt_tusktooth@reddit
insane, back in my day you couldnt mix and match graphics cards, is it different for AI stuff?
Special-Wolverine@reddit (OP)
Yes, different for AI stuff. You can even mix and match 30 series and 40 series, etc...
wheres__my__towel@reddit
Does this also apply for mixing RTX with data-center cards like V100s?
Scared_Astronaut9377@reddit
You have data security requirements higher than every military/intelligence/financial institution in the world, and you are solving it by using home-grade hardware. This makes a lot of sense!
possiblyraspberries@reddit
You realize things like Copilot aren't in GCC High yet, right?
Scared_Astronaut9377@reddit
Op's data cannot touch the internet, so gcc high is irrelevant. They have data way too secure for gcc high, and yet they process it on a home-grade home-made-level hardware instead of a DGX in their data center. Do you realize how cartoonishly comical this is?
possiblyraspberries@reddit
My point is that there is plenty of data out there that doesn't have a cloud option for LLMs. None of my GCC High clients can, which covers the "military" part of what you mentioned. I can't speak to the specifics of other industries' requirements.
It might be a bit odd to process data about zoomie boomies for the military (or whatever sensitive data OP has) on a home-built local LLM machine, but there's no need to be an ass.
Scared_Astronaut9377@reddit
I cannot see how clouds are relevant. OP's data cannot touch the internet. Not "cannot be processed by uncertified third party providers", but "has to be transferred using physical carriers between facilities if required". I am sure you know this is an extreme requirement even for military clients. And I am sure you know what level of other requirements, including for hardware, any organization with such requirements for data protection has.
As for not needing to be an ass, it's an entirely different topic. Not sure why you started mentioning GCC high if your only point was to teach me manners.
Perfect-Campaign9551@reddit
How fast is the video encode? It must tear right through it
Special-Wolverine@reddit (OP)
Surprisingly, not significantly faster than a single 4090 with my i9-13900K. So don't build this kind of thing if you're looking for that. At least in topaz video AI. I know there's other programs for video processing and rendering linearly with extra GPUs though
Special-Wolverine@reddit (OP)
1600 watts is fine - Don't want to trip my office breakers
Cerebral_Zero@reddit
Where's your power supply?
Additional_Test_758@reddit
In this case, it's rear mounted and out of sight.
Cerebral_Zero@reddit
I should've known that before. I'm having a tired day. A better question is how many PSU units or what behemoth is powering 3 of those cards?
CharlieInkwell@reddit
How much would you estimate it cost you?
Owl-Tea555@reddit
No nvlink for 40 series cards, does this actually have a sizable performance boost that is worth it?
FaatmanSlim@reddit
Most AI/ML tools should be able to run in parallel without requiring NVLink. You may be thinking about non-AI 3D (e.g. Unreal Engine) or video editing tools (like DaVinci Resolve) which I believe do require NVLink, otherwise limited to 1 GPU during rendering.
Special-Wolverine@reddit (OP)
Correct. Depends on the program. Topaz video AI allows you to split amongst all the gpus
ThenExtension9196@reddit
Looks great. Can clean that up with some 24vhps but other than that it’s a beautiful rig.
InterstellarReddit@reddit
I thought he had supreme RTX cards at one point before catching my mistake and was like holy shit.
kkhachadur@reddit
Nice build tho, I think you coulda gotten a second psu. That vertical 4090 doesnt look too happy.
LexiStarAngel@reddit
what exactly are you using it for?
TheWebbster@reddit
That's a nice use of space. The radiator for the lower MSI is behind the upright founder edition card?
Special-Wolverine@reddit (OP)
Yep, really should have taken more pictures
Special-Wolverine@reddit (OP)
The office stays pretty cold and is not dusty at all, so it's not an issue really
aphelion83@reddit
Really nice. Super clean. Bummer about the case, wonder if it'll be an issue since a fans blowing out won't create much airflow "inside."