GLM-4.5-Air llama.cpp experiences?

Nice! I'll dump a bit of the context I have on the implications so forgive me if you know any of it. This is part of my learning process and keeps me motivated better than studying alone these days. I prefer this when I have the option! Just remember not to trust me too much heh like I said kid with a circular saw vibes. \- consumer DDR5 boards often don't get full bandwidth on four DIMMs but it's a little complex so just worth looking in to. I tried peeking and all I found was that there are people who have talked about memory controller issues on your board (not always a bad sign, good to have data points, didn't find any about bandwidth-limited workloads though) \- I \*\*just\*\* found out why you may not want to give every thread to your CPU backend. Most setups (especially my DDR4) will saturate the memory bandwidth before all the cores are fully occupied. [See here for more](https://www.reddit.com/r/LocalLLaMA/comments/1gdxmz2/comment/lu5z5v3/), but I think you may actually be close to being able to use all your cores with 5200 DDR5. It depends on the memory controller, though. And you'll want to find a way to observe it saturating then experiment with backing off to do that properly. A little blind testing wouldn't hurt until then. \- You also have an edge in PCIe bw on gen 4 (the highest 3090 supports) but seeing as both of us are probably CPU bound running a 40-60GB model on 24GB+RAM I doubt it'l factor in here. \- with AVX512 you should def give ik\_llama.cpp a look. Happy to help anyone get it compiled and get a feel for how to find info in the repo which is optimized for ikawrakow's developer workflow to align with the purpose of the fork. \- Attention max-batch in particular should be useful as I think you'll be able to push your batch size with those 512-wide SIMD instructions on the CPU, thereby increasing your PCIe utilization (underutilized when CPU bound, remember) by sending more data per batch. \- When you're tight on VRAM that's important because it affects the size of the compute buffer and can be tuned for higher perf. The higher batch sizes will blow that compute buffer up without it and then you lose speed again by offloading more to CPU... there has to be a sweet spot. \- There is obviously a level of understanding which algorithms are going to run going on when he does this, but at [the end of this comment](https://github.com/ikawrakow/ik_llama.cpp/pull/237#issuecomment-2691104270) the author does some optimization on Deepseek Mini that opened my eyes. Obviously calculating the size of the matrix based on tokens per expert and the size of the token embeddings is doable but I don't yet know if I could determine the optimal size for my hardware without testing. If you want to do that math the values you need to plug in are prefixed with `glm4moe` in the model details.

Bottlenecking does funny things! Add in the way modern GGUFs use drastically different quantizations for different layers and I can see that happening. Like once it's bad it won't get worse until you have to start mmap'ing the weights off storage. Picture a bucket with a tiny hole and then a bucket 10/20/30 percent larger with the same sized hole, but don't get too attached to the analogy in detail. It breaks down. I can vouch for the ubergarm IQ2\_KL being better than the UD-IQ3\_K\_XL from unsloth for my workload. I wouldn't say it's fair to stack up against some of the other IQ2 quants I've seen (just comparing the layer sacrifices). My gut says we can totally bump those numbers up with your specs. If you like koboldcpp check out [croco.cpp](https://github.com/Nexesenex/croco.cpp) which used to be the "frankenfork". It's the ik\_llama.cpp equivalent down to supporting (currently) ik-only quants. You'll need to figure out which you can use but as CPU bound hybrid inference-ers we need all the CPU optimizations we can get -- that's ik's killer feature. There's also a guy who doesn't post quants, he posts a collection of all the layers quantized in all the ways that make sense. Then, using his tool, you download them and build a custom GGUF. I think ultimately this is going to be THE way to squeeze power out of frugal hardware. Current quants are jack of all trades. Oh and THANK YOU. I just re-ran with 8 threads to try to give you a datapoint and it's up to 10t/s tg. Looks like that stuff about memory bandwidth per core really matters. This is potentially very interesting if you can disable cores to save power as that often leads to MORE SPEED as you'll throttle less in long workloads.

Ah thanks! The llama.cpp cpu-moe flag is funnily enough just “ffn_.*_exps=CPU” and if you add the number of layers it just limits it. Not a terrible solution but also just a starting point!

Here’s an easy and free one. They are not kidding about turning off hyper threading (SMT for me on AMD) Holy shit. Night and day difference. Whole new baseline before I rebuild the OS.