"Cheap" 24GB GPU options for fine-tuning?
Posted by deus119@reddit | LocalLLaMA | View on Reddit | 18 comments
I'm currently weighing up options for a GPU to fine-tune larger LLMs (Deepseek 70b), as well as give me reasonable performance in inference. I'm willing to compromise speed for card capacity.
Was initially considering a 3090 but after some digging there seems to be a lot more NVIDIA cards that have potential (p40, ect) but I'm a little overwhelmed.
Endercraft2007@reddit
You want to make sure that you buy a card that is Turing or newer gen chip so modern CUDA is supported. If you would like to only run models not requiring CUDA then you can look at AMD cards too
FOE-tan@reddit
For non-CUDA, you could also wait for the Intel Pro B60 to launch later in the year, which apparently offers 24GB GDDR VRAM for $500 if they stick to MSRP.
Endercraft2007@reddit
Yes but in that case it would be easier to just buy a used 3090 IMO(depends on regin prices)
PaluMacil@reddit
I’ve never seen a used 3090 under $900 that I can recall. I haven’t looked in a while but have become convinced that people who say they are cheap haven’t looked in two years themselves.
mj3815@reddit
On the east coast, i have bought 2 3090s at $500 each and one at $700 all in the past 6 months. The first 2 from FB marketplace and the latter from reddit hardware swap.
PaluMacil@reddit
I hadn't thought to look for GPUs on FB Marketplace. Thanks for the tip
mj3815@reddit
It’s a bit of a pain, but have your guard up and do your due diligence and walk away if something feels off
PVPicker@reddit
Zotac is selling refurbished 3090s for $764 right now:
https://www.zotacstore.com/us/zt-a30900j-10p-r
90 day warranty, should be long enough to verify it's not defective. If it breaks after that, then that's just unfortunate luck, same as with any card.
PaluMacil@reddit
I hadn’t heard of them but decided to buy it. Unfortunately they won’t ship to Texas I guess. 🤷♂️ I’ll need to find another seller
Endercraft2007@reddit
It depends on region as I said.
PaluMacil@reddit
I’ve only looked at eBay. Do some regions have stores that sell used computer parts?
Endercraft2007@reddit
In my region, Serbia for example there is a site cslled kupujemprodajem where people post their stuff for sale or what they want to buy. I am sure there are simmular sites in other regions.
BackgroundAmoebaNine@reddit
Man it’s so cool that we all like ai from different parts of the Globe. It feels like we are so close even though we are so far, working on the same thing and sharing good knowledge.
PaluMacil@reddit
😎 cool… I suppose I won’t move just to enjoy that site, 😄 but I like to learn things and know the different dynamics in other places anyway
CheatCodesOfLife@reddit
P40 would be a pain for FT due to lack of BF16
What have you got now? You can run a 16GB T4 in google colab for free
FullstackSensei@reddit
For fine-tuning, nothing comes close to the price-performance of the 3090. The P40 is great for inference if you can find them for 200 or less. For 300 they're too expensive for the compute and memory bandwidth they have.
FullstackSensei@reddit
For fine-tuning, nothing comes close to the price-performance of the 3090. The P40 is great for inference if you can find them for 200 or less. For 300 they're too expensive for the compute and memory bandwidth they have.
DeltaSqueezer@reddit
I'd start with cloud rented to see how that works for you before committing to buying hardware.