On a Ryzen 9950X and optimized settings I get around 3 t/s (at q6_k) in more or less pure CPU performance for Qwen 235B, so a use epyc of a similar-ish generation on a DDR5 platform you'd expect to be about 6x the speed or so on the low end.

Obviously, less powerful servers or DDR4 servers (used xeons, older epycs, etc) you'd expect to get proportionally less (maybe 2x what I get?).

The other thing though, is that Qwen 3 235B uses *a lot* of raw memory. At q8 it's around 235GB of memory just for the weights (around 260GB for any appreciable context), and at q4 it's around half that.

The thing is, though, it's an MoE so only about \~20B parameters are active.

So, you have *a lot* of very "easy to calculate" parameters, if you will.

On the other hand, GPUs have very little memory, for the same price (an RTX 4090, for instance, has around 24GB of memory), but their memory is *very fast* and they have a lot of raw compute. I think the 4090 is over 1 TB/s of memory bandwidth, for example.

So, a GPU is sort of the opposite of what you'd want for running MoE models (for single-user inference).

On the other hand, a CPU has a lot of total memory, but not as much bandwidth, so it's a tradeoff.

I've found in my experience that it's *really easy* to trade off memory capacity for other things. You can use speculative decoding to run faster, or you can do crazy batching, or any other number of tricks to get more out of your system, but if you don't have enough memory, you can make it work but it sucks way worse.

Everyone has different preferences, though, and some people like to just throw as many GPUs as they can into a rig because it "just works". Things like DIGITS, or AMD Strix Halo mini PCs, and Apple Mac Studios are really nice because they don't use a lot of power and offer fairly good performance, but they are a bit pricey for what you get.