Help with MI50 and llama.cpp/ROCm 7.2

Posted by WhatererBlah555@reddit | LocalLLaMA | View on Reddit | 7 comments

I have an MI50 that I use with llama.cpp/Vulkan, however some models run quite slowly, so I'd like to try the ROCm backend, but no matter what I try it doesn't work.
Downloading the missing files from ArchLinux package doesn't work.
Downloading the files from coutryboycomputers doesn't work either
Building RocBLAS also doesn't work.

I'm not a Linux newbie, but at the same time I don't have days to spend on this issue... if someone has a guide that actually works to make llama.cpp run an MI50 with ROCm - ideally 7.2 - I would appreciate that.

[-]

brahh85@reddit

this is the one i tried
https://www.reddit.com/r/LocalLLaMA/comments/1pkvc85/comment/ntysctk/?context=3

this is another that should work
https://www.reddit.com/r/LocalLLaMA/comments/1s8thlo/build_script_for_llamacpp_for_rocm_including_mi50/

this is a comparison of rocm and vulkan with mi50
https://www.reddit.com/r/LocalLLaMA/comments/1rmt315/2x_mi50_32gb_quant_speed_comparison_version_2/

what command are you using for llamacpp?

what quant is your model?

[-]

WhatererBlah555@reddit (OP)

there are still a couple of things I need to figure out exactly, but this https://www.reddit.com/r/LocalLLaMA/comments/1s8thlo/build_script_for_llamacpp_for_rocm_including_mi50/ seems to work.

Thanks.

[-]

juss-i@reddit

There are nightly build tarballs for the "preview" versions 7.12 and 7.13. AMD's version numbering is weird btw.

Check out https://github.com/ROCm/TheRock/blob/main/RELEASES.md#installing-from-tarballs
and
https://rocm.nightlies.amd.com/tarball/ (filter for gfx906).

I grabbed one of the 7.12 nightlies a while ago, and it's been working fine. The only issue is that the tarball doesn't have an installer, so it's a bit more work to get all the binaries and libs in your path.

[-]

Material-Duck-6252@reddit

I have installed ROCm 6.3.3 (last official supported version for gfx906) and compiled llama.cpp on it. This served as a stable version for work. I would install ROCm 7.2, llama.cpp of ROCm 7.2 and vllm using docker for testing. [Mixa's version](mixa3607/ML-gfx906: ML software (llama.cpp, ComfyUI, vLLM) builds for AMD gfx906 GPUs, e.g. Radeon VII / MI50 / MI60) works well.

[-]