What do I miss with 7900 xtx today

[-]

TeakTop@reddit

I have an XTX, I use it with llama.cpp daily, and have done image generation in the past with SDXL. Most issues I've had is software not supporting AMD, or having to wait for updates. Hardware wise I would not say it's comparable to 4090 other than VRAM. In speed it's closer to a 4080. That said I have no complaints about speed for anything I've done.

Reply

[-]

JakoDel@reddit

how do you guys have so much money to spend yet dont do any research afterwards? llama.cpp pretty much halves the performance of amd cards lol

Reply

[-]

iamlazyboy@reddit

Honestly, for me running LLMs is quite new and not the main use for my 7900xtx, I mainly play games and recently started experimenting with local LLM as a "how can I use my GPU for something else than games while still being a fun experience?" Thing

Reply

[-]

JakoDel@reddit

it's one thing being new, another is settling on the worst option without any further research and also suggesting it to others. why did you reply when your situations are so different?

Reply

[-]

iamlazyboy@reddit

I'm sorry but I don't think I follow your train of thought here, op asked for those who have a 7900xtx their experience using it with LLM and you wanted me not to answer them as I started playing with LLM with my 7900XTX and didn't get any significant problem with it? I don't claim to be the most knowledgeable guy out there but it seems like the one who shouldn't be in the comments of this post between us is you imho as you seem to hate AMD and can't visibly comprehend people having a non Nvidia card and be happy with it and it's performances

Reply

[-]

JakoDel@reddit

I was talking about the difference between you, a new user, and the user I replied to, who has ultimately settled on something that halves the performance of AMD cards after months of use. where did I ever express my hate AMD? are you 14? you guys are involountarily damaging AMD, by letting people run their cards at half the speed with a project, llama.cpp, that doesnt care much about companies other than Nvidia. ultimately leading those that notice the sheer difference between their AMD and other nvidia cards to switch lmao. but as I said, you're new so thats all good, but I expect people that spent $900 on a card to do a tad more research before settling for months on the first thing they found. if that actually applies to you too, then I'm sorry but thats my opinion. exllamav2 literally doubles the performance on amd.

Reply

[-]

iamlazyboy@reddit

Oh ok, my bad I misunderstood your messages, I thought at first that your comment about having do much money to spend and yet settling to sub-par tools were about AMD being often viewed as not worth the investment when it comes to LLMs and AI in general to begin with, so I kept on thinking you were badmouthing AMD GPUs as a whole not really realizing it was mainly against llama.ccp's poor performance WITH AMD, English is my second language and sometimes my brain just decides to misunderstanding stuff, so I apologize for the misunderstanding and thanks for the clarification!

Reply

[-]

TeakTop@reddit

Yeah I also use it for gaming. Never heard of llama.cpp being slow on AMD. I build it with ROCm and never had any issue with speed.

Reply

[-]

iamlazyboy@reddit

I'm new to all that stuff so I'm still learning stuff but the only problem I had was ollama loading the model on my normal RAM but not in my gpu's VRAM despite inferring on my XTX for some reasons which was a bit limiting and frustrating bc I only have 16GB of ram, so I had to switch software and luckily LMstudio works like a charm, and as I understood, they both run llama.cpp under the hood (at least I think lm studio does) and don't have much problems with it, but can't really compare to the other sides as I'm not made in money and the XTX is good enough for what little AI I use it for

Reply

[-]

LoafyLemon@reddit

3090 is much faster during inference compared to a 7900 XTX so that checks out.

Reply

[-]

hainesk@reddit

When I look at inference speeds compared to my 7900xtx, it's only about 10%-15% faster.

Reply

[-]

LoafyLemon@reddit

RTX 3090 is 42% faster during inference. https://www.reddit.com/r/LocalLLaMA/comments/1ghvwsj/llamacpp_compute_and_memory_bandwidth_efficiency/

Reply

[-]

randomfoo2@reddit

The difference used to be smaller, but when llama.cpp added CUDA graph + Flash Attention, the gap really opened up (there are actually some potential ROCm llama.cpp optimizations available, but no maintainer: [https://github.com/ggerganov/llama.cpp/pull/7011](https://github.com/ggerganov/llama.cpp/pull/7011) ) - the 7900 XTX has both more theoretical FLOPS and MBW than the 3090 so in theory, it could be tuned to be much faster, but in practice...

Reply

[-]

iamlazyboy@reddit

I have the 7900XTX as well, I'm still messing around and trying some stuff and different sized models and like you, I don't really have big problems, especially now that ROCm is on windows, for the inference speed, I only have my XTX and don't really know the speed difference between it and the 4080-4090 and I'm mostly a gamer, so according of what I know (akak gaming perfs, idk how it can translate to inference speed) it's as fast as the 4080 in most games that doesn't require RT so the XTX being around this level of inference speed doesn't surprise me if what you say is true

Reply

[-]

randomfoo2@reddit

On my Ubuntu 22.04 LTS HWE system, my headless 7900 XTX idles at 14W. My idle RTX 3090 is 30W (it might have a headless ghost HDMI plug as it goes into a low-power mode without anything plugged in), my RTX 4090 is 8W. You can control PLs for both AMD and Nvidia cards so max power usage is sort of what you want to make it. You can see some recent threads on efficiency for token generation vs power (on the Nvidia side you can easily lop off 100W and still have 97% performance). vLLM works, you can see my doc [https://llm-tracker.info/howto/AMD-GPUs](https://llm-tracker.info/howto/AMD-GPUs) for a recently updated review of what works on RDNA3 as of last month. For inference the 7900 XTX can be an option, but I'll note that in my most [recent testing](https://www.reddit.com/r/LocalLLaMA/comments/1ghvwsj/llamacpp_compute_and_memory_bandwidth_efficiency/) (just posted), you can see in llama.cpp, the 7900 XTX significantly slower in both pp and tg vs the RTX 3090 atm the 3090 is 1.9X faster at prompt processing, and 1.6X faster at token generation, while being 1.2X the price. Especially if you think you're going to do training or use CUDA for other reasons (eg, there are a few interesting things like CTranslate2, ktransformers, thunder kittens, etc that are CUDA only)then you'd probably be better off with a 3090. (For inference-only, I wouldn't bother with/consider the 4090 at all.)

Reply

[-]

kryptkpr@reddit

Very unrelated to AMD but your comment made me go back and check on ctranslate2 and the devs have woken back up!! I'm gonna go try gemma2, it's Tok/sec performance with every inference engine i try is so bad compared to other models it's size I hope CT2 can deliver like it did with starcoder back in the day

Reply

[-]

randomfoo2@reddit

Two interesting things about CTranslate2. I saw arlo-phoenix has a hipified CTranslate2-rocm fork with partial support: [https://github.com/arlo-phoenix/CTranslate2-rocm](https://github.com/arlo-phoenix/CTranslate2-rocm) See discussion here: [https://github.com/OpenNMT/CTranslate2/issues/1072](https://github.com/OpenNMT/CTranslate2/issues/1072) CTranslate2 leverages oneDNN for some operations. One thing that is interesting is that it may be possible to actually use the mainline for some tasks? I saw this writeup the other day: [CTranslate2: Efficient Inference with Transformer Models on AMD GPUs](https://rocm.blogs.amd.com/artificial-intelligence/ctranslate2/README.html)

Reply

[-]

ILickMetalCans@reddit

I can't speak for much, but my 7900xtx does a good job for what I use it for. I usually do koboldcpp with 22b models in the 14-40gb range, and it's workable. 14gb being 10-20tok/s and 40 being around 1-2tok/s. No real fiddling done on my end and works well for chat bots and some work based stuff. I'm sure someone who actually knows what their doing could optimize my setup better. If you just want to run small models(13b and under), it's a great card for the price, and also game performance being good doesn't hurt.

Reply

[-]

AmazinglyObliviouse@reddit

Flux training was broken due to some missing parts for the attention operation I heard.

Reply

[-]

nero10578@reddit

Flash attention is a hack on ROCM

Reply

[-]

Thrumpwart@reddit

Official FA2 support was introduced in ROCM literally 2 or 3 days ago.

Reply

[-]

akimbra@reddit

Amd gpus are cheap if you don't value your time. The setup and making things work decently together requires a lot of time and more intricate knowledge. Get a 3090 if you're looking for a bang for buck setup. The t/s are good for most use cases. You can also get them at about 700EUR in the BENELUX area. If you want fast inference, just go with the latest nvidia X090 or X090 TI as the new 50 series will be released early next year. If you want more vram and don't care about money, get an A6000. It's a bit slower than the 4090 in inference, but you can run higher quantization and larger models on it. Though this is a bit overkill, IMHO.

Reply

[-]

FullstackSensei@reddit

I actually got 3 FEa for 500-550€ a pop. You need to hunt a bit, but it's pretty worth the savings if you're not using them for work

Reply

[-]

LoafyLemon@reddit

This so much. AMD is NOT ready for deployment in any meaningful way just yet. ROCm is a joke.

Reply

[-]

PraxisOG@reddit

I'm a pretty casual user, and have two 6800s for inference(chatbotting) and it's been great for the price. LM Studio lets me run 70b models at reading speed, but amd cards tend to be a bit slower than nvidia counterparts. Image gen stuff works too. I've been getting more into local llm stuff and want to fine tune some models, and supposedly qlora fine tuning works in Linux or WSL with ROCM.

Reply

[-]

badabimbadabum2@reddit (OP)

6800s is laptop card, and 8GB how you fit 70B?

Reply

[-]

Nyghtbynger@reddit

I believe he pluraled it

Reply

[-]

badabimbadabum2@reddit (OP)

true

Reply

[-]

PraxisOG@reddit

I just meant two rx 6800 cards. Total of 32 gb vram with ~1tb of total bandwidth runs llama 3.1 70b @ iq3xxs (~28gb) at 10 tok/s, though longer context drops that down by a few.

Reply

[-]

Nyghtbynger@reddit

I believe he pluraled it

Reply

[-]

b3081a@reddit

When zero monitors attached it saves a few additional watts, more like 15-20w, but this depends on aspm/runtime pm config/VRAM vendor etc, YMMV. vLLM works on a single Radeon GPU, but it does not idle properly from my experience. If you open nvtop or other GPU monitoring software it will idle at 100% utilization and max clock (\~3GHz), though only consuming around 100W of power and does not affect other apps' performance. llama.cpp has nice idle power, but when launching multiple llama.cpp processes it stops being so after about 2-3 instances, and creates a similar situation to vLLM. Some new quantization methods of llama.cpp (like iq\*) creates a lot of compute demand so when doing inference it's running at full power. For older ones like q4\_0/q8\_0, it's a lot less than that.

Reply

[-]

LoafyLemon@reddit

3090 idles at 8 watts sooo

Reply

[-]

Taronyuuu@reddit

My 3090 idles around 18 watts, I did research about this and it seems to depend on the brand. My 4060 idles around 4 watts.

Reply

[-]

MAY911@reddit

Can't say anything about the 7900xtx specifically but i am using a 6800xt and all things work easily for me. I installed rocm and use koboldcpp-rocm and things just work really. Don't know about image generation outside of koboldcpp-rocm but that works too without any problems so far. I am also thinking about getting a 7900xtx as they get cheaper but i am waiting what happens with the new generation from nvidia and the potential refresh of amd cards.

Reply

[-]

charmander_cha@reddit

Maybe your time but it's not rocket science.

Reply

[-]

Final-Rush759@reddit

Need to spend more time to make things work. Personally, I don't think it's worth it. It's hack. ROCM still hacks into Nvidia Cuda. I don't think Nvidia wants that to be a smooth experience. They may find a way to use it without Cuda. Until then, I won't touch it.

Reply

[-]

emprahsFury@reddit

You dont lose anything really. You will lose fast xformers, fast fast attention 2. But you do have them.

Reply

[-]

badabimbadabum2@reddit (OP)

Do I lose fast attention 2? [https://www.amd.com/en/resources/support-articles/release-notes/RN-AMDGPU-UNIFIED-LINUX-24-10-3-ROCM-6-2-3.html](https://www.amd.com/en/resources/support-articles/release-notes/RN-AMDGPU-UNIFIED-LINUX-24-10-3-ROCM-6-2-3.html)

Reply to Post

39 Comments