Ohh. 🤔 Okay ‼️ But what if we look at AMD Mi100 instinct,⁉️🙄 I can get it for $1000.
Posted by sub_RedditTor@reddit | LocalLLaMA | View on Reddit | 13 comments
Isn't memory bandwidth the king . ⁉️💪🤠☝️ Maybe fine tuned backends which can utilise the AI pro 9700 hardware will work better. 🧐
Ulterior-Motive_@reddit
I swear by them personally. The only trouble is no display outputs. I have to waste a PCIe slot that I could use for another MI100 on a crappy 2GB card because the motherboard won't boot without some kind of display out.
LevianMcBirdo@reddit
Could you go m.2 to pice? If you just need display out this should be enough, right?
Ulterior-Motive_@reddit
In theory, but the way the m.2 slots in my motherboard are laid out, it's difficult to find a m.2 to pcie adapter with a cable long and thin enough to work without the other GPUs getting in the way.
LevianMcBirdo@reddit
Aren't there Displaylink adapters that run through USB only? (Only would recommend that if you only need to look some very static information like once a day)
Willing_Landscape_61@reddit
What is the fine tuning situation with AMD Mi100 instinct ? How do they compare to 4090 ? What could I do with 4 or 8 of them compared to 4090s performance wise?
Eden1506@reddit
If its just for llms sure, but doing stuff like video generation, image generation, 3D model generation and many more require cuda or at the very least are quite troublesome to get working at even half the speed of a nvidia gpu simple because the majority of software was made with cuda in mind.
sub_RedditTor@reddit (OP)
Yes. But if you're good with configuring Rocm or Vulcan . Yes it's harder and sometimes not every LLM backend will work but for a specific case and if it works , I'd go with cheaper AMD .
HorribleMistake24@reddit
you got any pointers with Vulkan?
sub_RedditTor@reddit (OP)
Amm. Here's this. 🙄 https://github.com/GPUOpen-Drivers/AMDVLK
HorribleMistake24@reddit
Thanks, I’ve got ChatGPT talking me through getting it all running with an AMD card. I got it working, then ChatGPT got stuck in a loop where it was just giving me the same code snippet into some file in the Vulkan SDK.
I’ll figure it out eventually
sub_RedditTor@reddit (OP)
Try grok with Google Gemini. Chat Gi- ppity is one of the worst when it comes to hallucinations and rate rate limiting.
If you're using the free version, most likely the context window is full . https://github.com/vectara/hallucination-leaderboard
HorribleMistake24@reddit
i'm using chatgpt plus. once i get this shit figure out, what model do you recommend starting with?
sub_RedditTor@reddit (OP)
Hmm. What's your set-up because not all back-ends will support all Ai models ..
What do you have in mind or what you'll be using the local LLM for .?
The easiest set-up using Vulcan and AMD cards will be LM studio.