High-VRAM GPUS for us nerds.

Posted by arthurwolf@reddit | LocalLLaMA | View on Reddit | 96 comments

There are currently no (reasonably priced) graphics cards with a lot of VRAM (>= 64GB) to run large models. My expectation is, at some point, some manufacturer will make those happen. But I'm wondering if we (as a community) can make it happen sooner. VRAM is not that expensive (https://www.tomshardware.com/news/gddr6-vram-prices-plummet), so something like a 1060 with 64 or 128GB of RAM shouldn't be **too** expensive. Unless there is some technical reason this can't be done cheaply (or at all) that I'm missing, please enlighten my naive ass. Personally, if I'm going to put 900 euros into a graphics card, I'd rather it has fewer CUDA cores than a 3090 but more RAM than a 3090. Not sure about others here. Here are some solutions I can imagine: ## 1. Harass large manufacturers. If we all collectively email (or social-media-spam) large manufacturers of GPUs / graphics card, we might get them to understand there is a significant demand for these cards, and push them to release a product. ## 2. Get a smaller manufacturer to do a Kickstarter. Maybe we could find a smaller manufacturer of graphics card to understand this demand exists, and motivate them to get into that niche. They'd potentially do a Kickstarter for the board, so there wouldn't be too much of an upfront cost for them. And we as a community would be able to help/put our money where our mouth is. ##3. Get an Open-Source project started. Maybe we could find somebody who has already done some kind of graphics-card / advanced board as an open-source project, and motivate them to design this board for us. Maybe we can support them through some kind of donation thing as they do the work, and/or they can do a Kickstarter to finance the design and the early production. Maybe that person is on this sub, maybe that person is you? An option here for the Open-Source project, would be to use old/outdated GPUs/VRAM that is being sold at a discount, which would enable for a cheaper board (with lesser token-per-second, but still allowing us to run models we normally wouldn't be able to run). Any other ideas of how to get this off the ground? (a number 4 in this list?) Any recommendations of whom to contact for each of the 3 categories? Any reason why this is a terrible idea? Would **you** be interested in such a board? Thanks a lot in advance. Cheers.