Prompt processing is important, but exactly how much depends on the workload. For something like a chat bot with an unchanging history and incremental increases in token inputs, kv caching is going to save you tons of time and you only have to process the new prompts as they happen, and that is still very fast. But, if you have a workload where large prompts are provided and/or changed, then it will hurt badly, because it's just additional waiting time where absolutely nothing tangible is produced and you can't do anything. Interactive coding and RAG context filling are both examples of where this can happen.

On the other hand, I haven't looked up the actual compute specs on Granite Rapids. While I have no doubt it will do fine in token generation if it has enough cores, if the new instructions don't provide enough performance or if libraries don't take advantage of them, then it will be no faster than an M-series chip because the memory bandwidth is comparitively unimportant during that phase.

And as for the GPUs, I'm primarily talking about flexibility. You can always add GPUs later and spread workloads across them to increase performance. It's not ideal, but it is possible. Or, you can look at one of these crazy setups where people just put the money into used 3090s and have as many of them as possible. You aren't going to build a 500GB inference machine with 3090s (or at least you aren't going to do that sanely), but you could build a smaller one. I saw a 16x 3090 setup on Reddit the other day! It may or may not be a good idea, but it is possible. On a Mac, it isn't.

And then there's the power usage. The Mac is going to be efficient and small. All of this is kind of wacky, but if a small business or extreme hobbyist is set on experimenting with these kinds of things without going out and trying to purchase a DGX rack, all of these options are viable to a point, and they all have tradeoffs. Having some amount of capability in a very small, very quiet machine is something.