Koboldcpp appreciation post (image gen on AMD)

Posted by MixtureOfAmateurs@reddit | LocalLLaMA | View on Reddit | 21 comments

They've ([LostRuins](https://github.com/LostRuins) and [0cc4m](https://github.com/0cc4m)) just added stable diffusion support to Vulkan so now all my AMD brethren can enjoy fast LLM inference, image generation, and a nicer UI with the new Corpo theme. 1.3s/it is fine for an rx 6600 running SDXL, but is anyone running a 7600xt? If so please share your results and do a happy dance with me in celebration of Vulkan. It's so easy too! One executable!!!

Reply to Post

21 Comments

[-]

SquashFront1303@reddit

Me with an APU 😭

[-]

henk717@reddit

On my AMD APU it at least halved the image gen times, 5 seconds per iteration on SD1.5 is still better than 10 haha.

[-]

SquashFront1303@reddit

How are the images ? Are they any good?

[-]

henk717@reddit

As good as I would expect from raw image generations with those models. Of course if you have a dedicated image generation suite with post processing, controlnet stuff, etc you can get better results. But for a simple image model generator its giving the results I would expect.

[-]

SquashFront1303@reddit

I have tried koboldcpp with sd1.5 but the images are reddish i am running it with vulkun it is taking approx 5 seconds per init on eular a I don't know what i am doing wrong

[-]

henk717@reddit

For me that doesn't happen on either my Nvidia system or my AMD APU so this will be guess work for me on how to fix it. If you are quantizing the model try it without doing that, some models are sensitive to this and no longer produce correct results if you do. You may also want to try a different model, I tested anithing v30 (Not to be confused with Anything V3) and that one produces correct results (Unquantized).

[-]

Admirable-Star7088@reddit

Koboldcpp is a fantastic app and the team behind it do very valuable work for the LLM community. ❤️

[-]

paryska99@reddit

Yes! These guys are carrying us so hard. My steam deck is running lls thanks to them :D

[-]

LicensedTerrapin@reddit

Really? Like what?

[-]

paryska99@reddit

You mean models? Anything koboldcpp will run. Llama3.1 8B worked alright.

[-]

LicensedTerrapin@reddit

How fast is that on the steam deck?

[-]

paryska99@reddit

About 20-30t/s prompt processing and if i remember correctly around 4/5t/s generation

[-]

whoisraiden@reddit

That generation speed is really good for steam deck. Which quant did you use?

[-]

LippyBumblebutt@reddit

Deck has a quad channel memory interface with ~100GB/s memory bandwidth. That is quite good for an APU.

[-]

whoisraiden@reddit

Yeah I had no idea. It's perfectly sufficient.

[-]

paryska99@reddit

Just a regular q4

[-]

zaxwashere@reddit

neat, it let me run with vulkan on my 3060 and 6900xt, splitting vram across the two. does it make any sense to do that? uhhhh.... downloading a 20gb model now to test tonight, wonder if it'll be faster than cpu offloading or not. Regardless, really cool stuff!

[-]

YearZero@reddit

My favorite backend/front-end. I use the back-end with API to create some web-apps for specific uses. Like I got one that takes in a prompt and a csv file with a single column. Then it applies the prompt to every entry in that column and adds its response into the adjacent column. Great for data classification tasks, etc. The front-end is perfect for me - no setup, one .exe file, runs all GGUF's, has a beautiful interface, and every possible model parameter is exposed in settings to configure as needed. No other backend/frontend ticks all the boxes for me like this one, and I'm excited with every release.

[-]

Koboldcpp appreciation post (image gen on AMD)

Reply to Post

21 Comments

SquashFront1303@reddit

henk717@reddit

SquashFront1303@reddit

henk717@reddit

SquashFront1303@reddit

henk717@reddit

Admirable-Star7088@reddit

paryska99@reddit

LicensedTerrapin@reddit

paryska99@reddit

LicensedTerrapin@reddit

paryska99@reddit

whoisraiden@reddit

LippyBumblebutt@reddit

whoisraiden@reddit

paryska99@reddit

zaxwashere@reddit

YearZero@reddit

Downtown-Case-1755@reddit

henk717@reddit

Downtown-Case-1755@reddit