Have Anyone Successfully Run the New MiniCPM-o-4_5-gguf?

Posted by Iory1998@reddit | LocalLLaMA | View on Reddit | 14 comments

Hi, I saw yesterdary Openbmb adding this new model to HF. Link: [https://huggingface.co/openbmb/MiniCPM-o-4\_5-gguf](https://huggingface.co/openbmb/MiniCPM-o-4_5-gguf) It's an omni model that comes with vision and audio adaptors. I am wondering if anyone have successfully run it locally, and if so, how did you manage to do it?

Reply to Post

14 Comments

[-]

1lifebqrious@reddit

Works after setting up on my Mac Studio but voice glitches big time - even with 2000ms delay and 1024 MaxKV. I had change quite a bit of code in python around hardcoded cuda usage. May be I broke something doing those changes :D But, I guess the real issue is that PyTorch FlashAttention can't run on MPS and fall back algo is super old and slow. Anyone getting good duplex voice convo on Mac without constant voice glitches? If so, did you use llama.cpp?

[-]

Iory1998@reddit (OP)

It's not supported by llama.cpp yet 🥲

[-]

Interpause@reddit

got it working but had to fix it up quite a bit lol. but it really is super low latency

[-]

SOCSChamp@reddit

The webRTC demo? Mines coming through broken in chunks

[-]

Interpause@reddit

yeah, that happens if the speech inference cant keep up with realtime

[-]

Ryoonya@reddit

Do you have a guide to get it working?

[-]

Interpause@reddit

easiest to use ubuntu and follow their tutorial. i think half the problems i ran into was cause they assumed it was an ubuntu system and im on cachyos. and don't do anything funny like me and set custom cmake args

[-]

CystralSkye@reddit

Can you please reveal how you got around to making it work?

[-]

Klutzy-Snow8016@reddit

They have this demo: [https://github.com/OpenSQZ/MiniCPM-V-CookBook/blob/main/demo/web\_demo/WebRTC\_Demo/README.md](https://github.com/OpenSQZ/MiniCPM-V-CookBook/blob/main/demo/web_demo/WebRTC_Demo/README.md) It says it's designed for Macs, but it also works on Linux. You basically just have to compile llama.cpp-omni for CUDA and set CUDA\_VISIBLE\_DEVICES=0 at runtime. To access it from another machine, you might have to use an ssh tunnel so that the traffic appears to come from localhost. They said on X that they are working on a more plug-and-play implementation of the demo, too.

[-]

Iory1998@reddit (OP)

Linux! I use windows. What a shame.

[-]

Klutzy-Snow8016@reddit

I don't see any reason why it wouldn't work in Windows. It's Python code, a couple of Docker containers, and a fork of llama.cpp.

[-]

Iory1998@reddit (OP)

Oh I see. Thank you very much. I'll try to see if I can make it work.

[-]

overand@reddit

WSL2 is the way to go with this - just, before you try to get *this* working, make sure you're able to run CUDA stuff in WSL2 (and in docker in WSL2, if that's relevant to this - I'm not sure.) Google stuff like "Test CUDA in Docker" to get quick ways to verify that your GPU will work for linux docker containers

[-]

martinerous@reddit

It might even work with WSL2. Some time ago I got VoxCPM (quite nice TTS, easy to finetune for new languages) working on WSL2 with nano-vllm.