Experts-Volunteers needed for Vulkan on ik_llama.cpp

Posted by pmttyji@reddit | LocalLLaMA | View on Reddit | 12 comments

ik_llama.cpp is great for both CPU & CUDA. Need legends to make Vulkan better as well.

https://github.com/ikawrakow/ik_llama.cpp/discussions/590#discussioncomment-16357564

So, after bringing the Vulkan back-end up to speed some time ago, I felt that I simply don't have the bandwidth to also maintain it. In llama.cpp there are two maintainers who do nothing else but Vulkan.
But if you are willing to do that, we can try to resurrect Vulkan. Of particular interest would be to implement the graph parallel stuff in the Vulkan back-end (after porting quite a few missing ops that have accumulated since my last effort).
I guess, the issue will be that I'm a complete beginner when it comes to Vulkan. So, unlike your CPU changes prepared with the help of Claude where I was able to quickly spot a problem, with Vulkan we will be left at Claude's mercy, which may turn into a complete disaster with time. So, I think, if you want to become a Vulkan maintainer for ik_llama.cpp, you need to become significantly more knowledgable than me.

https://github.com/ikawrakow/ik_llama.cpp/pull/608

https://github.com/ikawrakow/ik_llama.cpp/discussions/562

Thanks in advance!

[-]

milpster@reddit

The maintainer is toxic. I wouldn't even write another bug report let alone contribute to that project.

LegacyRemaster@reddit

It's a real shame that Vulkan doesn't work well on ikllama. Using RTX 6000 + 2x W7800 I can't take advantage of ikllama since I only get 2 tokens/sec with Vulkan. I hope someone can help

fallingdowndizzyvr@reddit

It's a real shame that there's a civil war in "llama.cpp". Remember, they were once the same project. Now even if third parties try to port features of one to the another, it gets denied.

crantob@reddit

Not always is one path the best.

Not always do we know which potential path is the best.

Do you want one government bureau in charge of all makings of shoes as well?

Yeah, that's why the best path is when people work together to combine their best ideas.

agree but.. it is what it is.

tomByrer@reddit

It is the Linux distro splintering all over again...

Remove_Ayys@reddit

Any efforts put towards ik_llama.cpp are wasted.

pmttyji@reddit (OP)

Why? It's good to have another hybrid(CPU+Vulkan).

I had some quick look at mainline's vulkan shaders and CUDA kernels, they looks quite different...
I guess Vulkan lacks many features comparing to CUDA, so performance may not be as good as you want...