Hardware costs to run 90B llama at home?
Posted by Sea-Replacement7541@reddit | LocalLLaMA | View on Reddit | 146 comments
- Speed doesn’t need to be chatgpt fast.
- Only text generation. No vision, fine tuning etc.
- No api calls, completely offline.
I doubt I will be able to afford it. But want to dream a bit.
Rough, shoot from the hip-number?
baehyunsol@reddit
afaik, llama 3.1 70b and llama 3.2 90b have the same text model. you don't need 90b if you're only using chat
Gokudomatic@reddit
And in the case of a 70b model, what would OP need to buy?
kiselsa@reddit
Option 1(very cheap): 64gb ram - run Q5km with 1t/s on CPU
Option 2 (mid): 2x P40 (2x 24gb VRAM) - run q4km with 6-7 t/s, llama.cpp only
Option 3(top): 2x 3090/4090 (2x 24gb vRAM) - run exl2 with exllamav2 - faster prompt processing and generation, also can train diffusion and text models.
No-Refrigerator-1672@reddit
Used P40 right now costs $300 and up. I think the times when P40 was a good option are long gone now.
crpto42069@reddit
This is one of the craziest investments I've ever heard of. EOL graphics hardware going up in value.
No-Refrigerator-1672@reddit
I remember looking to buy P40s a year ago for less than $100 a piece, and I didn't do it just because I didn't feel like making dedicated AI home server back then. Now I regret that decision.
fallingdowndizzyvr@reddit
MI25s were $65 when I got one. I heard they were as low as $45. The new value champ is the P102 10GB for $40.
No-Refrigerator-1672@reddit
P102-100 has good perfomance, but abismol vram. Unusable for any model beyond 8B. Mi25 now goes for just a little above $100, but this particular GPU somehow is not officially supported by ROCm and need tinkering to make it work. MI60 right now can give 32GB VRAM at respectable inference speed for just under $300, which is a good all-round option, but, from what I've been able to find up on the internet, is higly problematic when building multi-GPU setup. Mi50 is also a very respectable cheap option if you can feet all your need into 16 gb vram.
fallingdowndizzyvr@reddit
Can't you just edit the supported models list and recompile?
Regardless, that's why people either flash it to a Vega or into a WX9100. Then it will be supported. If you flash it to WX9100 that will enable it's lone DP port and you can actually use it as a GPU.
No-Refrigerator-1672@reddit
1) As per your own link, Gemma 16B Q4 takes 8.8GB VRAM. That leaves a little more than 1GB for context. That means your model won't be able to handle more than 1-2k of tokens. That's really not much, this model won't be able to handle anything besides a short conversation. No document Q&A, no API calling, no summarization, no web search, basically nothing out of actually useful stuff will be available to you.
2) Yes, that's precisely what other people do to make them work, and that's precisely why it's bad - cause you have to hack it, and the moment you do, your config differs forom qnything covered by docs and standarts, and if you have any troubles nobody will be able to consult you.
SwordsAndElectrons@reddit
I don't really get the love for P102-100s either.
Is it possible to cost effectively use, say, 4-6 of them? I guess I just don't see where a single unit is all that exciting.
explorigin@reddit
Not really no. You need a motherboard and power supply that can handle 4-6 cards.
No-Refrigerator-1672@reddit
When you parallelize, LLM over multiple GPUs, you need to keep context on all of them. So it is possible to use 8x P102 to run inference, but you won't get 8x usable VRAM thos way, you'll hit diminishint returns very quickly.
However, P102 is an incredibly good offer if you want to enhance your rig with Whisper, AI TTS, or Stable diffusion, as those models fit into 10GB VRAM pretty easily.
kiselsa@reddit
I bought one for 90$ and I regret not buying more
No-Refrigerator-1672@reddit
Well, given how volatile is hardware market is moving now, and also guessing that soon some custom silicon designed exclusively for AI will emerge, I can assure you that this regret is temporary. In two years or so we will have the opportunity to regret not buying Tesla V100s at low price, and then maybe feel the same about Tesla T4s or T40s.
ladz@reddit
This. Everyone is working on it. These chips will be cheap as chips in a few years.
thegreatcerebral@reddit
I doubt they will (want) to keep up with demand so they can demand a higher price due to keeping supply low which will also keep demand skyrocketing. Think like when the Nintendo Wii came out and Nintendo just withheld stock for like a year and half so that everyone and their grandparents would buy one because everyone was going crazy. ...same thing here.
The ONLY way they would actually meet or exceed the demand would be if they found a way to turn it into a subscription model where if you didn't keep your subscription then it ran slower or not at all kind of thing. Then they would magically be able to produce them until their nuts fell off and everyone could buy one for $10.
spacetech3000@reddit
Everyone working on this is just nvidia.
Original_Finding2212@reddit
Cerebras, Hailo, Groq beg to differ
kiselsa@reddit
Well, it's still noticably cheaper than 3090, in my country at least.
No-Refrigerator-1672@reddit
Now Tesla M40 became a budget-friendly option. Also 24GB Vram, price on ebay under $100, and arpund 30 tok/s on 8B Q4 model if you manage to cool it down. Sure, it's slower, less eficient, and less supported than P40, but still works.
Caffdy@reddit
Jesus, we're scrapping the bottom of the barrel, huh
No-Refrigerator-1672@reddit
Well, that's where the cheap path leads now. I.e. I'm located in EU; here, the only way to get your hads on Teslas that I've found is either importing them from US, or from China. That means paying for overseas shipping and taxes. So P40 to me is luke 450 eur, while used 3090 in my local town is like 550 eur, with some options of 500 eur popping up occasionally. It's a no-brainer to not even consider P40 for EU citizen.
Nyghtbynger@reddit
I have opportunities for 450€ 3090 in my city
No-Refrigerator-1672@reddit
Which country is it? I'm in Latvia and for 450eur I can get 3080Ti at best, never saw 3090 to go this low here.
Nyghtbynger@reddit
France, hand to hand market
kiselsa@reddit
Probably, but it isn't supported anymore by latest drivers. With my P40 I just installed latest studio driver and everything worked perfectly out of the box. With m40, you need to look for older builds, etc.
Also m40 is two cards glued together, so memory used for context is doubled.
No-Refrigerator-1672@reddit
M40 is single chip card. You're mixing it up either with K80, which is a dual chip acnient tech, or M10, which is 4x laptop GPUs.
Cyber-exe@reddit
We should start advising nobody spend over 300 for any of those P40's. You're often paying extra to add a shroud and fan to it, and maybe PCIe to EPS power adapters too. It's just extra factors beyond the original price tag.
FencingNerd@reddit
$300 is a bargain for a 24GB card. 4060Ti is $450 for 16GB. 3090 is like $1200+.
Cyber-exe@reddit
It's not simple as 24gb card. The card is old, support is weak, no warranty. You get a much weaker amount of compute power for that VRAM compared to other models. The lack of a display out port is one factor that stops casual gamers from picking up one of these and then the cooling situation is another factor.
fallingdowndizzyvr@reddit
I rather get two 3060 for that same $300. Not only does it end up being cheaper, no having to get cooling solutions, it works better.
LeBoulu777@reddit
It's what I bough for my new computer setup for $450 CAD,.
330d@reddit
I'm seeing many 3090 on ebay.com in $700 range
fallingdowndizzyvr@reddit
That's only from small time resellers pushing up the price. Wait for a major liquidator to well... liquidate another batch.
Or you can shop around on AE where you can still find them for about half that price. Of course, there are plenty of gougers on AE as well selling them for as much as $800.
skrshawk@reddit
I got in on my pair while they were $200 apiece. If I replace them with 3090s I'd definitely resell them for a profit.
No-Refrigerator-1672@reddit
I think (or rather hope) that if Nvidia decides not to screw us around and make RTX50 a better perfomance per $, then the whole market will have to readjust down and this will brind up P40s too. Not to $100 level, but still. However, when I remember the launch pricing of RTX40, I do realize that Nvidia providing better value in new gen is highly unlikely.
skrshawk@reddit
Indeed, there is zero market pressure for them to drop prices. They're the only game in town in an industry that people are throwing venture capital around like candy. The only potential price limit is on what cloud providers can rent the things out for over what period of time. They're also definitely not going to want to undercut their cloud providers by selling hardware that can do it locally for a better value. From their point of view, the open community isn't needed to advance their margins.
UpperParamedicDude@reddit
A few months ago i bought one for $220 when people already claimed that it became too expensive
delicatemicdrop@reddit
I got an open box 3090 for this reason and have I believe 64gb RAM. I get pretty decent speeds and it works for me for just roleplay. My PC is about a year old now and it also plays games very well and I think it was definitely worth the investment.
kiselsa@reddit
Are you running 70b models? If yes, you probably use offloading half of layers to CPU with llama.cpp or run very low quant (iq2_xxs).
Offloading with llama.cpp unfortunately drastically slows down prompt processing so you can't really roleplay in group chats (whole context should be swapped to avoid personality leak)
delicatemicdrop@reddit
I use kobold and I can check and see what my speeds are, tbh don't know them offhand. I use about 12k context. Re: what quant, I don't use group chats, so that's a fair assessment. Midnight-Miqu-70B-v1.5.Q2_K.gguf is what I run so yes, it is a smaller quant. My default layers to CPU is 58 layers to GPU, rest offloaded.
It's not the BEST setup by any means, but it serves my purposes well while also running my games well. Everyone has different needs/uses for their LLMs so what works for me and was a good investment and should continue to meet my needs for at least another year or two.
Mart-McUH@reddit
Offloading slows mostly inference. Prompt processing does not change that much, it seems to be done on GPU either way (eg when I do prompt then 4090 is working, 4060Ti is completely idle - 0% - despite part of model is there - and CPU is <5%, so mostly housekeeping tasks I guess). What slows prompt processing most is increasing context size.
Here some numbers for L3.1 70B with 4090 + 4060 Ti + DDR5 RAM (when offloaded) with 8k context and 512 BS prompt processing time:
IQ3_M (81/81 so fully loaded) - 16 sec.
IQ4_XS (74/81 loaded) - 27s (slower but also the model is bigger)
Now, increasing to 12k context
IQ3_M (80/81 loaded) - 48s (huge slowdown but mostly because of bigger context)
Btw with 4090 + DDR5 (before I got 4060Ti) you can run IQ3_M 8k context with 55/81 on GPU and get \~3T/s inference. Prompt process was 24s (so yes, bit slower than those 16s when it is fully loaded but not that dramatic).
kiselsa@reddit
Problem is, if I can offload model fully to gpu, then I can use exl2 that has much faster prompt processing than gguf.
Caffdy@reddit
Group chats? How does that work?
kiselsa@reddit
Group chats in sillytavern. You can rp with multiple characters. When character talks, character card at the beginning of the chat is swapped.
Caffdy@reddit
Didn't know of that, do yoy have a tutorial in hand I can check out?
kiselsa@reddit
Well, you can find sillytavern docs on group chats or some YouTube video.
It's intuitive though and you can figure out how to use them without tutorials.
Apprehensive_Path465@reddit
You can also assemble a server on AMD EPYC 7002. The motherboards have 8-channel DDR4 memory. On a system with one CPUs, you can run Llama 70B 8Q with 2 t/s.
On a system with two CPUs and 512GB of RAM (16 channels), you can even run Llama 405B 8Q (\~0.6 t/s).
Willing_Landscape_61@reddit
"two CPUs and 512GB of RAM (16 channels), you can even run Llama 405B Q8 (~0.6 t/s)." I'm tempted to do just that. Do you have any references? I presume it only depends on RAM speed. Which RAM speed would give you such 0.6 t/s ?
Thx.
Apprehensive_Path465@reddit
I bought this configuration about a year ago on aliexpress, but the links don't work anymore.
MB (MZ72-HB0) + 2xCPU (7502) cost about $1700. Memory - Micron MTA36ASF4G72PZ-3G2.
But I used some old 2933 MHz modules that I already had, so the frequency is not 3200, but 2933.
fairydreaming@reddit
Can you share some llama.cpp performance values for using 1 CPU vs using 2 CPUs?
Apprehensive_Path465@reddit
I don't know how to correctly disable one processor for a clean comparison, but I can say that llama.cpp with the "--numa distribute" key on a dual-processor system speeds up inference by about 1.5 times.
fairydreaming@reddit
This is just a guess, but I think you would have to examine
/proc/cpuinfo, get allprocessornumbers (ideally a continuous range) belonging to samephysical id(for example equal to 0) and then prefixllama-clicommand withwhere
num_fromandnum_tois the identified range ofprocessornumbers.I'm asking because one person who tried llama.cpp on dual Epyc Genoa system reported that it worked faster on a single CPU (yeah I know it's weird, maybe it's related to limited bandwidth between CPUs).
dodo13333@reddit
For server configuration, you have to check a qualified vendor list (QVL) to determine compatible RAM. Servers run on EEC RAM. You have to check motherboard manufacturer site for QVL.
not_as_smart@reddit
What motherboard and CPU would you suggest for this?
0xd00d@reddit
it's nice but seems too slow for practicality, DDR5 12 channel has to get in the range of affordability sometime soon...
Prince_Harming_You@reddit
“DDR5 12 channel has to get in the range of affordability sometime soon”
No, it doesn’t, since it’s literally the absolute fastest non-‘unified’ system RAM configuration
Each 64g DIMM of DDR5 4800 (ECC) is like $500x12=$6000 just for RAM, not including the like $8,000-$15,000 for the rest of the hardware only to end up with something that has half the memory bandwidth of a 4090/Apple M3 Max
I like the optimism, but it will be a while
0xd00d@reddit
apple's apple silicon chip naming scheme is the most well thought out naming scheme in decades I feel like.
I am pretty sure new higher bandwidth memory tech is dropping, very soon, and CXL is coming for greatly enhanced capacity. What I'm saying is 500 or so GB/s is enough bandwidth to do some cooking with, whereas 200 or so GB/s off DDR4 8 channel is workable but kind of painful. I'll concede though, that it's nothing to shake a stick at.
Yeah i guess will still take a while. oh well.
Prince_Harming_You@reddit
Lol now that I think about it you're right, better than Intel with their European sedan style naming scheme Core Ultra 7 265K
Extreme would be so swag
Apprehensive_Path465@reddit
llama.cpp supports multiprocessor systems. With the "--numa distribute" key, performance increases by about 1.5 times.
Original_Finding2212@reddit
Wouldn’t a used M2 Mac do?
kiselsa@reddit
Well, if you can find a Mac with 64GB of RAM at a reasonable price.
because as far as I know it will be more expensive than 2x 3090, and 3090 is much more powerful and has more applications.
Perfect-Campaign9551@reddit
Can already run 70b on 3090 with additional 32g system ram, around 1.5t/s
TwiKing@reddit
Does the error/hallucination rate happen more on Cpu than Gpu? Or are they the same?
droid786@reddit
I have a very stupid question, how do you calculate the compute requirement for training & building RAG Apps and inference?
infectus_@reddit
!remindme 1 day
TheDreamWoken@reddit
Or get a A6000
Ready_Safety_9587@reddit
Good comment thanks
MoffKalast@reddit
Option (overly cheap): 32GB RAM - run IQ2_XS with partial offloading
vir_db@reddit
I run it on and i5 3th gen with 32GB of RAM and a RTX 3060 12 GB. Is pretty slow but it works.
Gokudomatic@reddit
It looks like it's more about memory than performance. I have an i7 7th gen with only 16GB and an 1060 GTX with only 6GB, and llama3.1:8b model is the best I can run decently.
MLDataScientist@reddit
Another option: 2x AMD MI60 which gives OP 64 GB VRAM for $600. They OP needs to buy a used PC that has two pcie x16 slots. Total could be around $1000 and you would get around 9 tokens/s for 70B models.
Cyber-exe@reddit
That could be a good option if I make a dedicated AI rig
GradatimRecovery@reddit
OP can use his existing motherboard with a M.2 to PCI-E adapter
Durian881@reddit
I'm using Apple M2 Max with 30 GPUs and 64GB ram. MLX 4 bit 70B runs at ~7 tokens per second.
robogame_dev@reddit
Good datapoint - As described that's $2400 new.. probably OP's best deal if they can work on mac, as well as in a week or two new M4s are expected and price new and used may go down...
erick-fear@reddit
Running llama 3.1 70b on CPU only 4vcpu, 42gb of ram. That I would consider minimum.
CandyFromABaby91@reddit
A MacBook with 64GB ram
e79683074@reddit
128GB of DDR5 RAM should be somewhat cheap these days
No-Refrigerator-1672@reddit
It's better to choose ddr4, as it's extremely cheap ($1.5/gb for shoddy chinese brands and $2/gb for low-end SKUs of reputable brands) and you can leverage the dropping prices on the used market. CPU inference is painfully slow regadless of what ram you have, so why pay more?
FunnyAsparagus1253@reddit
I couldn’t handle CPU inference of a 13b model past bare 1 question 1 answer, and 7b was painfully slow once the context got up a bit. Fine for a novelty but no fun at all for chatting
ProlixOCs@reddit
How was it this slow? I was getting 2.9-3.1 tok/s using dual 2697v4s and 192GB DDR4-1866 ECC on Noromaid-13B (91-94GB/s mem bandwidth and 36 threads assigned to Ollama)
0xd00d@reddit
sweet data point, thank you for sharing. that is still too slow a rate for many uses but it's approaching usable. tok/s will drop precipitously to run models large enough to actually take advantage of that amount of memory however.
I guess that DDR4-1866 is running in quad channel mode. Do you need both CPUs to achieve quad channel on that system?
ProlixOCs@reddit
Just the one CPU is necessary for quad channel (would be limited to 68-71GB/s due to channel/rank/QPI interleave), but I’m running a Penguin Relion 2900 and this is a 24x8GB setup. A 22B model like Trinity-Codestral-22B runs about 1.7-2T/s, not the fastest but not too bad either.
FunnyAsparagus1253@reddit
Well looks like your system is better than mine was. I gave up when it hit 10 mins until first token never mind watching them agonisingly tick out probably about 0.5 t/s 😅
petuman@reddit
at $2 you're at consumer DDR5 prices already -- GSkill sells 96GB kit for $190 https://pcpartpicker.com/product/rYkH99/gskill-flare-x5-96-gb-2-x-48-gb-ddr5-5200-cl40-memory-f5-5200j4040a48gx2-fx5
e79683074@reddit
Incorrect.
You are bound by RAM bandwidth. DDR4 bandwidth is much lower than DDR5. You pay more (not even that more) to have higher speeds.
We are talking like half speed or something like that for DDR4
Cressio@reddit
Could you elaborate on that last part? Haven’t heard of that. Does it apply to DDR4 too?
Inkbot_dev@reddit
The memory controllers on consumer chips can only handle full speed with single rank ram populated in 2 slots. 96gb is the largest you can use (2x 48gb) if you want your ram to run full speed with an XMP profile.
ggone20@reddit
Other than being able to be offline, running an LLM ‘at home’ will never be cost effective - just use together.ai or some other hosted service and pay $0.2-1.80 per million tokens in/out (depends on model). If you go out and spend several thousand dollars on hardware to run a 70b model with no quantization (if you’re going to quant something… don’t. It’s never worth it - you’re not running a large model on small hardware, you’re giving the model brain damage so it can fit.
You need roughly 280GB VRAM to run a 70b at full precision. $30k minimum? Plus the technical skills to do parallel inference.
Just use an API for most things and run LLaMA3.1 8B locally if needed ‘offline’.
0xd00d@reddit
It's fair to keep in mind that lowering precision impacts quality but afaik for almost any practical purpose, going from the standard "base" fp16 down to fp8 shows next to no quality loss. A little bit more down to 6 bits is still decent and sure as hell lets you fit into commodity hardware a lot easier...
The vast majority of us do not have the luxury of running stuff at full precision. Especially when there is no need to.
A lot of us have solar with some surplus capacity which make the electricity free, and tinkering with hardware is a reward unto itself. The point is solid about API (it's indispensable for getting work done at this point these days...) but for some things where privacy matters we want to be able to self-host.
Mythril_Zombie@reddit
Is there one that allows you to use your own model? I couldn't find anything in the together site that would, and they didn't seem to have that many.
Am I missing something there?
It said you could train them, is that how you'd do it?
Sorry, I'm new to a lot of this.
ggone20@reddit
They have tons of models. Do you actually have a CUSTOM model to use? If you just mean some open source model by ‘your own model’ then they have LLaMa 3.1&2 (text-only 2 for now) as well as many others (but why would you use anything else as they are currently the ‘best’ at most things).
As far as a place that will allow you to host a fully custom model - that’s like runpod or several other GPU rental providers. I believe it’s approximately $4-6/hr last I checked to rent enough compute to host a 70/72B model.
That said, the OP asked about self-hosting - so assuming they mean ‘just’ an open model. Which brings me back to: why not just use the API ok together or groq or many other inference providers?
I_can_see_threw_time@reddit
for just text, you should use llama 3.1 70B, same thing, no difference in evaluation results from 3.2 90B
4 bit quant is probably as low as you'd like to go, awq maybe, or exl2 5.0 bpw (some options here to play with)
this is something like 35-40 GB (V)RAM + context
You "can" run this with like 48 GB of regular ram on a regular PC.
It will take a long time. [memory bandwidth of ram] / [model size in GB] = \~20 / 44 so something like half a token per second for generation, and that doesn't take into account reading the initial prompt ingestion.
I think speed does matter somewhat, as you'd likely get bored of this toy pretty quickly if you are waiting for minutes for responses if you are doing chat.
I'd probably go for (2) 3090 cards, i think they are unfortunately like 700-800 used USD a piece now
That would get you 48 GB VRAM
To calculate max tokens per second, [memory bandwidth of 3090s] / [model size in GB] 936 GB/s / 44GB, so something like 20 tokens / second.
note sure of your budget.
building computers is a whole other discussion, but https://pcpartpicker.com/ can help guide whether things are compatible.
Not sure what you have for motherboard / cpu / etc, but you will also have to make sure you have room, (enough pcie slots, keeping in mind that ht 3090 is i think 3 slots wide), and a PSU that is big enough (I'd probably do overkill with 1500W , but that would add like 400-500), although you should be able to power limit the cards to match something smaller without a major hit on performance.
If this is a new build altogether, I'd probably look into a microcenter bundle deals, or really anything that is relatively recent. for inference the speed of the cpu and the ram doesn't really matter. Ideally have enough pci lanes for at least 4x, although that really only affects the load time of the model, not inference. with most motherboards it would be like 16x and 4x, but you might find one that has the ability to do 8x 8x 4x , but in that case you will likely need gpu riser cables and something like a open mining rig to hold it (although these are cheap), also creativity maybe, but keep in mind the 3090 are hot and need air
0xd00d@reddit
that's an astonishingly elegant rule of thumb i haven't actually seen until now. it captures very effectively how the way it works is by flowing the entire model over itself repeatedly for every token, so the latency for generating one token is the time it takes to flow the entire model through the available bandwidth.
Herr_Drosselmeyer@reddit
I don't think there's a text only 90b version of LLama 3 (or 2 for that matter). At that size, there's only the model that includes vision. Text only models usually come in at 70b and then tend to jump past 100b.
Napkin math for the 90B model: you would need about 90GB of VRAM to run in 8 bit, roughly 45 to run in 4 bit. Since we need to add in a bit more for context and whatnot, let's make it 50.
This puts us in a bit of an awkward situation: if we go with a "budget" machine with two used 3090s, we'll be a few GB short and will have to go with a lower quant or split. Or we wait for the 5090, get two of those and then we can fit it comfortably. We can't feasibly run this model on one 3090 and expect it to be usable.
Since you specified text only though, let's look at 70b instead. With the same napkin math, we can fit a 70b at 4 bit into those two 3090s. With one 3090 we'd again have to go to a lower quant or split.
So, TLDR, you're looking at the following ballpark prices:
Single 3090 (used) + matching config at about $2,000 - can run 70b kinda ok, can't realistically run 90b
Dual 3090 (used) + matching config at about $ 3,000 - can run 70b decently, can run 90b kinda ok
Dual 5090 (new) + matching config at about $ 6,000 - can run 70b comfortably, can run 90b decently
CandyFromABaby91@reddit
A MacBook pro with an M3 Max and 64GB ram would work and is an easier setup.
Herr_Drosselmeyer@reddit
Quite possibly but I don't know jack about Macs so that's why I'm not mentioning them.
CandyFromABaby91@reddit
One thing to know is VRAM and system ram is shared. So it’s an easy way to get massive amounts of VRAM. It’s a cheat code for LLMs 😅
bobartig@reddit
While it certainly simplifies things greatly (I'm enjoying LM studio on my Macbook w/ 36GB ram) is it at all cost-effective? E.g. currently an Mac Studio M2 Ultra with 128GB RAM is just under $5,000. What's a similar PC setup? $2000? $10,000? I can't do GPU price math.
edude03@reddit
The m* are also fast-ish at inference so it’s not just getting 128gb of ram into a single box but also getting fast cards to compare apples to apples. And yeah 3x3090s used plus a server board and cpu is 5-7k depending on how lucky you are
GimmePanties@reddit
Also the electricity costs on a Mac are lower. A Max maxes out at 100W while a 3090 is 350W just for the card, add more for the rest of the machine.
0xd00d@reddit
tough to compare too, since splitting work across multiple 3090s will introduce a significant amount of overhead, but for stuff that fits, a 3090 is easily 3x faster, which *sorta* works to erase the 3x higher power consumption, obviously the mac being a self-contained device still winds up more efficient. But it is worth keeping in mind a GPU of this caliber has 3 times the memory bandwidth on tap.
FunnyAsparagus1253@reddit
Yeah but nobody runs multiple cards at full power here
GimmePanties@reddit
Oh? Enlighten me… is one card doing the work while the others are there for VRAM?
My experience of running multiple cards was 3 Radeon HD 6990s for Bitcoin mining in the early 2010s. Each card had dual GPUs and load was 365W per card, and those ran under full load continuously. Saved on heating, but electricity bill was insane.
FunnyAsparagus1253@reddit
Well what I’m led to believe is that during inference, the cards take turns to do the processing on their own chunks, plus, you can power limit them quite a lot for only a few % performance loss. I have my 250w P40s limited to 175w, for example. I’m not arguing with you about the mac being lower power, I’m just saying…
GimmePanties@reddit
Okay, yeah I was just interested. Maybe the layers are spread out across the cards.
This way of doing it probably better for the longevity of the GPUs. Miners burning them out was fairly common back then.
CandyFromABaby91@reddit
You don’t need an ultra. I’m running it on an M1 Pro.
robertotomas@reddit
He’s right. At 48gb you can only run a q3 with context of about 8k. 64gb would put q5 on the table , which is more important with llama 3.x since they quantize poorly, and longer context sizes
user258823@reddit
Llama-3.2-90B-Vision is literally just LLama-3.1-70B with vision attached to it, use Llama-3.1-70B instead if you don't want vision.
If speed really doesn't matter, then you can run anything even on the worst hardware with enough disk space.
For example, I managed to run Q2_K quantized Falcon-180B on 6 GB VRAM and 16 GB RAM with 256 GB pagefile at \~10 minutes per token.
GirthusThiccus@reddit
Good God, if we follow this line of thinking, we're gonna have to implement metrics of SSD wear and tear costs per complete sentence inferenced.
101m4n@reddit
Anything can be vram ram if you're ambitious enough 😁
RedKnightRG@reddit
Quoting LLMs in minutes per token is like when the military quotes M1 Abrams fuel economy in gallons per mile...
Weary_Long3409@reddit
I know 90B is great, but below 5 t/s on short context is awful. When 1-3 t/s on longer context, it just feels unusable and wasting time. Might be 70B still reasonable with 8 t/s.
Just out of context, the new qwen 2.5 32B is a good balance at the performance a bit better than gpt-4o-mini. Go the 72B will suffice at gpt-4o level.
TheKaitchup@reddit
Llama 3.2 90B is a vision model. It's essentially Llama 3.1 70B with a vision module on top of it. If you don't need vision, use Llama 3.1 70B. If you want a better model, use Qwen2.5 72B instead.
Roland_Bodel_the_2nd@reddit
Macbook Pro (or desktop Mac) with >90GB RAM. Rough cost $6k but it can be your primary computer also.
Cracked_sash@reddit
,,, 1
ttkciar@reddit
I use older dual-Xeon systems (T7910 with 2x E5-2660v3) with 256GB of RAM for CPU inference. They're quite slow at it, but only cost me $800.
You can probably find older single-Xeon systems with 128GB which work about as well for about $600.
quadgnim@reddit
Understand, the raw 70b text model is about 146GB. A 4090 card is around $2k new on Amazon. It only has 24GB. so 146/24=6.08, so figure 7 cards or $14k.
Most of these other solutions are using highly quantized models (think compression) which can affect the quality of the results from the raw model. However, depending on your use case, that might be perfectly fine. I run 8b quantized, and I'm happy. The 8B model is over 13GB raw, but quantized only 8GB.
Some older/slower cards also have 24GB, and some might even have 32 or 40GB. So If you really shop cards around, you can run the raw model on fewer/cheaper cards.
And if you're wondering, professional grade cards such as the H100 offer 80GB, so you can run 70B raw on just 2 cards. But, they can be upwards of $30k for a single card.
Have you considered using the cloud? AWS BedRock offers the raw 70b model for pennies per request. Looking real quick, it's $0.002 cents per 1000 tokens in and out each. So prompt cost and reply cost. I'd venture most requests prompts are well under 1000 tokens.
Lissanro@reddit
Price of 4090 does not matter, and for inference, size of un-quantized model does not matter either, it only matters for training.
Cost of 3090 is around $600 and its VRAM have comparable speed to 4090, so inference speed would be comparable too. And 70B model fits well on a pair of 3090 cards.
quadgnim@reddit
you aren't getting a 3090 for $600 unless its used and/or refirb. They're showing $1200-$1400 on amazon.
It's important to educate people on the difference between quantized and raw. They're NOT equal. You maybe can get good enough results with quantized, but they're not equal. That's why people who build quantized models offer many different variants, and why the original model builders keep them full size. So don't say it Doesn't matter. Maybe to you it doesn't matter, but the OP never explained what his use case was, other than performance wasn't so much a factor.
Downvoting me for trying to help is stupid, childish and imature. but I guess that's what the internet does, brings out the worst in people.
Lissanro@reddit
For the record, I did not downvote you. But my guess you got downvoted because almost nobody runs "raw" models for a good reason (except for training and generating quants based on them), and at 8bpw quality is practically equal to the unquantized version even for smaller models. Even transformer-based image producing models, generate results at Q6-Q8 practically not distinguishable from the FP16 reference, and generative text models are usually less sensitive to quantization than image focused ones, especially true for large models.
For example, tested with MMLU Pro Mistral Large 2 at 4bpw and 5bpw, with results almost equal, and on the level of the reference scores. Q4, Q6, Q8 and FP16 cache also produced nearly equal scores, ironically quantized cache sometimes produces slightly higher scores at Q6 and Q8, than at FP16, while Q4 lower only by very small margin. So I run Mistral Large 2 at 5bpw with Q6 cache, knowing for a fact that greater precision would change practically nothing.
But this is not even the most important factor - Mistral Large 2 (or fine-tunes based on it) at 4bpw with Q4 cache will work much better than Llama 70B at 8bpw with Q8 cache, both for creative writing and programming tasks, just because it has more parameters. Below 4bpw, it start to degrade quickly, at 3.5bpw it will not be as presice for programming tasks, at 3bpw it will be more suitable for creative writing than programming, and any lower than that, it may become worse than 70B model running at higher precision. The same is true for other models, 32B Qwen2.5 at 8bpw will be worse than 72B Qwen2.5 at 4bpw. Given OP specifically mentioned it may be hard for them to afford, it is safe to say that it is highly unlikely they will be able to buy more than 2-4 3090 cards, with a pair of used 3090 usually the best choice for running 70B models.
As of the price, it makes no sense to buy 3090 for twice its real cost. There are no benefit to it. I purchased two of my 3090 from trusted online store where I could return them if I do not like them within a week or two, and another 2 directly from original owners, running memtest_vulkan for about an hour before paying money to ensure there are no issues with VRAM (both in terms of no memory errors, and no overheating). If video card is not defective and can get through memtest_vulkan for a hour or more, it is extremely unlikely to fail within next few years. So even if Amazon promises extra warranty, it just not worth it if you can buy twice as much card for the same cost.
Mythril_Zombie@reddit
Is there a good resource somewhere that I can read about all this? All the models and numbers are just bewildering. Generating quants? Q cache?
How do you know what Qwen does compared to Mistral? I need a wiki.
G4M35@reddit
I asked Perplexity:
To run a 90B parameter Llama model at home, you'll need significant hardware resources. Here's an overview of the costs and options:
Hardware Requirements
The primary considerations for running a 90B Llama model are:
Cost Estimates
The cost can vary widely depending on the specific hardware configuration:
GPU-based Solutions
CPU-based Solutions
Alternative Options
Performance Considerations
It's important to note that these are rough estimates, and prices can fluctuate. Additionally, the hardware landscape is constantly evolving, with new options potentially becoming available in the future.
While running a 90B Llama model at home is feasible, it requires a significant investment. For many users, cloud-based solutions or smaller models might be more practical and cost-effective alternatives.
Citations: [1] https://www.reddit.com/r/LocalLLaMA/comments/1g3dtyy/hardware_costs_to_run_90b_llama_at_home/ [2] https://deepinfra.com [3] https://news.ycombinator.com/item?id=37067933 [4] https://news.ycombinator.com/item?id=35008694 [5] https://www.intel.com/content/www/us/en/developer/articles/technical/intel-ai-solutions-support-the-new-llama-3-2-model.html [6] https://newsroom.arm.com/news/ai-inference-everywhere-with-new-llama-llms-on-arm [7] https://cloud.google.com/vertex-ai/generative-ai/docs/partner-models/llama [8] https://cloud.google.com/vertex-ai/generative-ai/docs/open-models/use-llama
GoldWarlock@reddit
Any M Mac with enough memory to run the quant you want. 64gb will run Q4 I think.
ieat314@reddit
You can buy a workstation with two (2x) Xeon gold 6128 cpus each with 6 channels of memory so totaling 12 channels. 6128 supports 2666mhz that transfer 8 bytes per transfer meaning 21,328mb/s x 6 channels = 127,968mb/s per cpu so x 2 = 255,936mb/s or 255gb/s. Now fill those up with 2666mhz 8gb ecc sticks for 96gb of usable memory and could fit the 90gb model depending on compression. Using the correct compression for CPU could get you closer to the max throughput explained above. You will also have 12 more spots to fill in with 12 more 8gb sticks or go crazy and do 32gb or 64gb sticks, but the key will be to fill out the minimum 12 channels to max out the throughput.
So you could be looking at a 70b model, let’s just say you compress to get a straight 70gb then you do throughput 255gb divide by size 70gb and you get ~3.64 tokens per second for inference. Nothings perfect but you could optimize and with different compressions and configurations for CPU inference and probably get that number around 2.5-3 tks.
Cost of this would be less than the cost of a 3090 with great upgrade paths for adding GPUs in the future. You can look into HP, Dell, Lenovo workstations on eBay. I’m running a Dell 7920 due to the 1400w power supply, pcie lanes, front swap drive bays and other features that will allow me to build out a home server that can handle some of the best local llms out today with the option to run those big ones at above 1tks… and if plugged into a home assistant, you can command the assistant to work on a problem and it gets to it in the background so those lower speeds aren’t as noticeable. Plop a 3090 or other high memory bandwidth card in there for faster inference if you need, but it’s a good all around project machine that won’t break the bank AND if you start to hate AI stuff you still have a great home server that can act as a media server or (name your game) server or a home lab to mess around with. This allows you to sit and wait for deals on 3090 or other highly sought after cards.
The cons to this are the slowness, but as explained above for a 70gb 70b model you’re looking at 3tks which isn’t unbearable, you could work an agent system into the inference flow like a big little set up or consensus by comity etc that use smaller models do do most of the grunt work. Another con is hardware comparability. I was able to snag 2 P40s for under $300. I could not for the life of me get them to run in my dell 7090 workstation and I would say I am pretty versed in end user and server computer hardware, though I’ve found a person on Reddit that claimed the opposite. I sold them off for a profit since the prices went up I’m assuming due to people like me trying to build out budget local systems. Another compatibility is ram. With the old xeons you’re looking at 2666 or 2999 mhz depending on skew, and not all sticks will work with the cpu/mobo, I’d refer to the manufacturer and then resellers with lists of compatibility and then cross reference on eBay listing. I got 4 8gb sticks to add to the 8x8gb installed in the system I bought. I did all of the above and I sometimes can’t post because of memory issues, but reseating seems to fix it. Annoying but a con if you’re looking for easy set up like an all in one Mac system or building out a normal consumer hardware system and plopping in 3090s or better. Another similar solution would be new consumer CPUs with blazing fast DDR5. Even mini PCs will have your dual channel set up with blazing fast ddr5 you could probably push that 3tks with a system that is 5-10 years newer in terms of technology and will offer longer support (you risk not getting a feature that’s compatible with a cpu released in 2016-2016 - you see this with VINO I believe already on non scalable xeons)
knook@reddit
I'm also looking to spec out a build. Will models be able to share system Ram with Vram? In other words is it worth having a lot of ram if I still plan on running on a GPU like a P40?
Terminator857@reddit
$1,500 used on ebay: 3090 system with 64 gb of RAM and quantized to 5 bits.
GradatimRecovery@reddit
$600 for a pair of MI60 using 4-bit quantization https://www.reddit.com/r/LocalLLaMA/comments/1fxn8xf/comment/lqp62uh/
With a $3k MacBook you can do your ERP sitting in the corner of the cafe
skrshawk@reddit
I went searching for MI60s on eBay the other day, saw a listing for like 100 available, but now it's gone and I don't see any others around. In theory the MI100 around $1k is another viable option in the same price range at the 3090 since it has more VRAM, as long as the driver support is there. You'd still need to run it in a proper server or with cooling jank.
GradatimRecovery@reddit
In someone’s garage is twenty $100 mining boards, each with five MI60’s. Just kidding… I hope
skrshawk@reddit
Sad but likely true.
robogame_dev@reddit
$3K for a Mac Studio w/ M2 Max and 96GB memory would do you, maybe wait a week or two they may release M4 and prices may go down and/or used market may become better
Zeddi2892@reddit
Low Speed, Low Cost: Build PC with 128 GB RAM and modern CPU.
Mid Speed, Mid Cost: Wait for Macbook Pro M4 and buy the 128 GB Version
High Speed, High Cost: Build a VRAM Server and throw at least 5 3090s into it.
Highest Speed, Highest cost: Get a B200 for half a Million Dollars, this bad boy will run better than ChatGPT ;)
maxigs0@reddit
A runpod instance able to run it, will be maybe 3-4$ per hour. That's where i would start, if you want to play with it.
Building something for offline use, you are probably looking at 4-5000$ for Mac studio of self built system with enough (V)RAM. The latter might be slightly cheaper, but possibly faster and use a lot more power (that can be another 50ct/hr where i live).
If you have a lot of patience a 500$ kit of DDR5 memory would do the job.
Lissanro@reddit
Self-built system would be much cheaper. For 70B models, a pair of 3090 cards is enough, their cost is around $600 per card, total cost for the whole PC could be around $2K, and it will be much faster for inference than Mac too.
johakine@reddit
Patience is the king! I'd pay $3-4/hour for tests first.
krewenki@reddit
Vast also makes it cheap to run for a short period. Unless you’re running inference 24/7 for months it seems to be a lot more economical to rent the capacity
ozzeruk82@reddit
Cheapest method. Any PC with 64GB ram can run a quantised version of the Llama 3.1 70B model. It will be slow and frustrating, but it will work.
Nice method. Any PC with a RTX 3090 card ($500-750 second hand for the card). Will run a heavily quantised version but reasonable speed. I do this myself, pretty satisfying. Nicer still if you can use 2x3090s.
I would personally run Linux and ollama for simplicity, connecting via Open-WebUI from another PC elsewhere in the house or your phone.
All 100% offline, no cloud, nothing. Just needs electricity for the computers.
nachocdn@reddit
0ppppppppp
nero10579@reddit
4x3090 running 90B at 4-bit would be the ideal. Processing image takes way longer as they are essentially a lot if tokens so running stuff on CPU I don’t think is ideal unless you want to wait. 4x3090 machines that I’ve been building is about $6K.
sleepy_roger@reddit
For me I built a 2x3090 machine for around $1500 I had all the parts besides the 3090s and got the 3090s for around $1300. I've looked at a40 builds as well, could get a couple for $600.
2x4090's - $4,500 - $4,800 2x3090s - $1,800 - $2,000 2xA40's - $1,200 - $1,500
These are super ballpark numbers to give you an idea. The rest of the cost of course depends on mobo/cpu/ram/case/PSU, etc.
Rich_Repeat_22@reddit
To load 90B FP16/BF/16 you need 180GB VRAM/RAM + another 96 RAM to be safe.
a) You can get a 2x 64 epic Zen4 for around $2600 on ebay, or $2400 for a 96 core one. + RAM
b) Alternatives single MI300X and 128GB RAM in your PC. Cost around $15000 + your current PC (fastest option of all)
c) An Epic Zen3 server with enough DDR4 RAM.
d) An Epic Zen3 server with 6xMI100. You are looking something around $5500. Faster option than (d).
Upcoming : 2 x AMD AI 390 Strix Halo laptops with 128GB ram each having 96GB allocated in VRAM linked together. You are looking for something like $2500-$2600 for both.
synn89@reddit
Running 70-90B's at a decent speed and quant at home would be around 3-5k worth of hardware. You'd either want a dual 3090 build or a Mac M1/M2 Ultra 64-128GB(128 being preferred). The 3090's will be wanted if you want to do vision, training or image generation(Stable Diffusion/Flux). The Mac is better for pure inference as the 128GB will run at a higher quant, handle larger models, is very quiet and barely uses any power.
I have both setups and use my Mac M1 128GB for text inference pretty much exclusively.
jacek2023@reddit
I use 3090 for models up to 70B, I usually download ggufs of size around 40GB so half is on GPU and that acceptable speed for me, smallers models fit all on GPU