High-VRAM GPUS for us nerds.
Posted by arthurwolf@reddit | LocalLLaMA | View on Reddit | 96 comments
There are currently no (reasonably priced) graphics cards with a lot of VRAM (>= 64GB) to run large models.
My expectation is, at some point, some manufacturer will make those happen. But I'm wondering if we (as a community) can make it happen sooner.
VRAM is not that expensive (https://www.tomshardware.com/news/gddr6-vram-prices-plummet), so something like a 1060 with 64 or 128GB of RAM shouldn't be **too** expensive. Unless there is some technical reason this can't be done cheaply (or at all) that I'm missing, please enlighten my naive ass.
Personally, if I'm going to put 900 euros into a graphics card, I'd rather it has fewer CUDA cores than a 3090 but more RAM than a 3090. Not sure about others here.
Here are some solutions I can imagine:
## 1. Harass large manufacturers.
If we all collectively email (or social-media-spam) large manufacturers of GPUs / graphics card, we might get them to understand there is a significant demand for these cards, and push them to release a product.
## 2. Get a smaller manufacturer to do a Kickstarter.
Maybe we could find a smaller manufacturer of graphics card to understand this demand exists, and motivate them to get into that niche.
They'd potentially do a Kickstarter for the board, so there wouldn't be too much of an upfront cost for them. And we as a community would be able to help/put our money where our mouth is.
##3. Get an Open-Source project started.
Maybe we could find somebody who has already done some kind of graphics-card / advanced board as an open-source project, and motivate them to design this board for us. Maybe we can support them through some kind of donation thing as they do the work, and/or they can do a Kickstarter to finance the design and the early production.
Maybe that person is on this sub, maybe that person is you?
An option here for the Open-Source project, would be to use old/outdated GPUs/VRAM that is being sold at a discount, which would enable for a cheaper board (with lesser token-per-second, but still allowing us to run models we normally wouldn't be able to run).
Any other ideas of how to get this off the ground? (a number 4 in this list?)
Any recommendations of whom to contact for each of the 3 categories?
Any reason why this is a terrible idea?
Would **you** be interested in such a board?
Thanks a lot in advance.
Cheers.
96 Comments
mrxaviolo99@reddit
arthurwolf@reddit (OP)
mrxaviolo99@reddit
mrxaviolo99@reddit
Clamo636@reddit
fallingdowndizzyvr@reddit
Temporary-Advisor101@reddit
fallingdowndizzyvr@reddit
Temporary-Advisor101@reddit
fallingdowndizzyvr@reddit
AmericanNewt8@reddit
Entire-Branch7007@reddit
AmericanNewt8@reddit
deleted_by_reddit@reddit
dkarlovi@reddit
Affectionate-Mango19@reddit
PapaZiro@reddit
grizzlor_@reddit
greysourcecode@reddit
epicwisdom@reddit
greysourcecode@reddit
epicwisdom@reddit
noth606@reddit
Clamo636@reddit
Deadshot1583@reddit
Choice-Host-1599@reddit
PikaPikaDude@reddit
Cane_P@reddit
stevetheborg@reddit
stevetheborg@reddit
reidy-@reddit
onafoggynight@reddit
noth606@reddit
juwonpee@reddit
arthurwolf@reddit (OP)
BlandUnicorn@reddit
clockish@reddit
pstuart@reddit
Kooky_Arugula78@reddit
libertyraptor@reddit
Leading-Tone4615@reddit
SunsetQuest@reddit
mr___w@reddit
SunsetQuest@reddit
SunsetQuest@reddit
beedunc@reddit
nostriluu@reddit
anomaly256@reddit
nostriluu@reddit
nipple_salad_69@reddit
syle_is_here@reddit
Cyric_of_Waterdeep@reddit
osmarks@reddit
plasma_node@reddit
osmarks@reddit
plasma_node@reddit
SomeOddCodeGuy@reddit
Winter_Tension5432@reddit
hadlockkkkk@reddit
silenceimpaired@reddit
gigamiga@reddit
woadwarrior@reddit
LukaC99@reddit
Massive_Robot_Cactus@reddit
grizzlor_@reddit
grizzlor_@reddit
CryptoCryst828282@reddit
M34L@reddit
arthurwolf@reddit (OP)
burritolittledonkey@reddit
fallingdowndizzyvr@reddit
burritolittledonkey@reddit
noth606@reddit
Caderent@reddit
ThePizaMan@reddit
NeedRoaldDial37@reddit
arthurwolf@reddit (OP)
Agreeable-Worker7659@reddit
Level-Elk259@reddit
arthurwolf@reddit (OP)
Level-Elk259@reddit
arthurwolf@reddit (OP)
Level-Elk259@reddit
learnereverything@reddit
AndrewH73333@reddit
RapidInference9001@reddit
Temporary_Payment593@reddit
RapidInference9001@reddit
Dry_Parfait2606@reddit
Humble-Ad-1302@reddit
Conundrum1859@reddit
Omnic19@reddit
drivetinkerer@reddit
arthurwolf@reddit (OP)
CryptoCryst828282@reddit
arthurwolf@reddit (OP)