The mystery of Apple M3 Ultra GPU performance

Posted by Ok_Warning2146@reddit | LocalLLaMA | View on Reddit | 31 comments

The press release of M3 Ultra claimed that its GPU performance is 2.6x of M1 Ultra. [https://www.reddit.com/r/LocalLLaMA/comments/1j4jpij/comment/mgg62l5/?context=3](https://www.reddit.com/r/LocalLLaMA/comments/1j4jpij/comment/mgg62l5/?context=3) I thought the 2.6x gain was due to doubling of shader per core. But some people pointed out that the gain can be attributed to the existence of ray tracing cores in M3 that is not in M1/M2. To investigate the impact of ray tracing, I looked up the previous press release for M3. [https://www.apple.com/hk/en/newsroom/2023/10/apple-unveils-m3-m3-pro-and-m3-max-the-most-advanced-chips-for-a-personal-computer/](https://www.apple.com/hk/en/newsroom/2023/10/apple-unveils-m3-m3-pro-and-m3-max-the-most-advanced-chips-for-a-personal-computer/) It says M3 Max is 1.5x M1 Max, M3 Pro is 1.4x M1 Pro and M3 is 1.65x M1. Then constructed this table: |M3 vs M1|Ultra|Max|Pro|Vanilla| |:-|:-|:-|:-|:-| |GPU Gain|2.6x|1.5x|1.4x|1.65x| |M3 FP16|57.344|28.672|12.9024|7.168| |M1 FP16|42.5984|21.2992|10.6496|5.3248| |FP16 Gain|1.3462x|1.3462x|1.2132x|1.3462x| |RT Gain|1.9314x|1.1142x|1.154x|1.2257x| If I assume no doubling of shader per core, then M3 Ultra FP16 is 57.344. I assume the overall GPU gain is the product of FP16 and RT. Then I calculated the RT Gain and noticed that M3 Ultra's RT Gain is very different from the others. If I assume M3 Ultra FP16 is 114.688, then RT Gain is 0.9657x. This is closer to the other RT Gain but still a bit off. So the conclusion is that RT cores probably doesn't explain this 2.6x gain fully. Any ray tracing experts here that can share their opinion?

Reply to Post

31 Comments

[-]

Its_Powerful_Bonus@reddit

IMO more accurate estimation can be achieved by compare M1 Max and M1 Ultra, calculate real gain between those two (possibly also compare M2 Max vs M2 Ultra to double check the math) and then estimate gain between M3 Max vs M3 Ultra. Marketing data is as reliable as politicians’ declarations in election campaigns 🤣

[-]

Fresh_Yam169@reddit

Apple had a hard time synchronising M1 Ultra, on paper it was 2 Maxes, in reality it was way worse. M2 got better, but we don’t really know how M3 is doing without benchmarks.

[-]

Its_Powerful_Bonus@reddit

Exactly. This was my point. Let’s say M1 ultra is 1.6x faster in interface speed than M1 Max. M2 Ultra is 1.65x faster in interface speed than M2 Max. So M3 Ultra should be not worse than 1.65x speed of M3 Max. This logic is more reliable than estimation based on Apple marketing claims. Cheers!

[-]

Its_Powerful_Bonus@reddit

After 10 months from last comment nemotron-3-nano on M1 Ultra 48GPU: 66.54 tok/sec•2044 tokens•0.45s to first token M3 Max 40GPU: 80.90 tok/sec•2462 tokens•0.26s to first token

[-]

Ok_Warning2146@reddit (OP)

What do you mean by synchronising? For LLM, M2 Ultra is more than double the inference speed than M1 Max when the former's bandwidth is 819.2GB/s and the latter 409.6GB/s. [https://github.com/XiongjieDai/GPU-Benchmarks-on-LLM-Inference](https://github.com/XiongjieDai/GPU-Benchmarks-on-LLM-Inference)

[-]

Fresh_Yam169@reddit

In ideal conditions, M1 Ultra should yield 2x performance of M1 Max, but in reality you don’t have a single chip, you have two chips connected. When you have two CPUs and two GPUs, you are going to have a synchronisation problem if you want them to work as a single processor. It starts with booting both chips together (and offset of one millisecond matters) and ends with which one is managing that particular chunk of memory. You’re also going to have problems with one chip trying to work with memory managed by other chip (they have RAM onboard, which means one chip literally working with RAM in other chip, which brings additional latency (and boy, visiting RAM onboard is taking eternity compared to L1 cache). And synchronisation issue gets many times worse if you trying to stitch 4 chips together… well, that’s why M1 Extreme was never released. And if we are being honest, M1 Ultra obviously was a disaster should have never been released.

[-]

b0tbuilder@reddit

You don’t have two GPUs, you have two APUs connected by a silicon interposed.

[-]

Spanky2k@reddit

I'm guessing they mean that the M1 Ultra did not end up giving double the performance of the M1 Max in real world use cases back when it came out. However, that was before LLMs at home were really a thing and they scale incredibly well on these machines meaning that M1 Ultras *are* about double the performance of M1 Max chips.

[-]

b0tbuilder@reddit

Memory bandwidth increase.

[-]

Ok_Warning2146@reddit (OP)

No increase. It is the same 819.2GB as M2 Ultra and M1 Ultra.

[-]

cyboghostginx@reddit

Still doesn't support Cuda, waste of money

[-]

TechNerd10191@reddit

If you want to do inference only (or limited fine-tuning) Macs are fine - you don't have 2 buy 4x RTX 6000 Ada for $40k (not including a $5k CPU, $2k RAM, $2k Motherboard)

[-]

scapocchione@reddit

Inference only? I regularly finetune on my macs, no matter owning an A6000, which will be the last nvidia card I bought. That mentality of "no cuda = waste of money" is exactly what got us into a monopoly where I paid five grands for a 48gb, 768 Gb/s bw card. Now we can have 512gb (!), 819 Gb/s memory for the price of a single 6000 Ada card, and MLX works great.

[-]

TechNerd10191@reddit

How is fine-tuning on Macs compared to Nvidia GPUs? I haven't read much about it.

[-]

Zyj@reddit

What's the speed of inference on M1 Ultra 20 core these days? Can it take advantage of its memory bandwidth?

[-]

Ok_Warning2146@reddit (OP)

Should be similar to m3 ultra.

[-]

Zyj@reddit

Isn't it starved by lack of gpu power?

[-]

Ok_Warning2146@reddit (OP)

It can be. Even though RAM speed is the same. M3 Max is 80% faster than M1 Max for inference. [https://github.com/XiongjieDai/GPU-Benchmarks-on-LLM-Inference](https://github.com/XiongjieDai/GPU-Benchmarks-on-LLM-Inference)

[-]

Zyj@reddit

Perhaps the gap will close with better software

[-]

Abject_Radio4179@reddit

For me the real downer is the M3 Ultra 60C. It will have just 10% more VRAM bandwidth vs the M3 Max, which doesn’t justify the much greater price. Therefore, the only M3 Ultra worth consideration is the 80C.

[-]

Ok_Warning2146@reddit (OP)

M3 Ultra 60C only has 614.4GB/s. It is better to buy M4 Max that has 546GB/s.

[-]

Glittering-Fold-8499@reddit

Where did you see this? Apple website explicitly states 819GB/s for both 60C and 80C.

[-]

Ok_Warning2146@reddit (OP)

Sorry for the error. I suppose apple should be more authoritive. RAM speed is determined by number of memory controller only. So if reduction in cores doesn't reduce controller number then it should just be as fast.

[-]

Abject_Radio4179@reddit

Indeed. On the M1 and M2 the binned Ultra at least had the same RAM bandwidth as the fully enabled chip.

[-]

anzzax@reddit

Thanks you reminded that m3 max's are not equal.

[-]

poli-cya@reddit

I understand everyone is excited and hopeful, but this is reaching Anandtech forum levels of questionable hypothetical calculation. I thought your last post was likely far off same as the commenter you linked. Not intentionally misleading but so many best-case assumptions that it likely wouldn't reflect reality. Aren't we less than a week away from actual hardware? Rather than fumbling with numbers based on what appear to be heavily misleading benchmarks, might I suggest simply waiting?

[-]

JacketHistorical2321@reddit

Shall we bring digits into the convo? It's been weeks without an update to the actual bandwidth specs and people in this form keep talking about how it'll probably beat this or that in terms of performance. You can't even pre-order the damn thing

[-]

Disk-Alarming@reddit

Your chart is wrong. 1.6X M1 Pro is not the same as 1.6X M1 Max. The GPU in the M1 Pro is not the same as the GPU in the M1 Max. That is where your confusion in numbers is. The Base M3 is 1.6X M1 Pro.

[-]

Ok_Warning2146@reddit (OP)

I was trying to gauge the effect of ray tracing cores. I find the effect is similar among Max, Pro and base but not Ultra. That's what I was trying to do.

[-]

Spanky2k@reddit

I would ignore anything Apple has said in terms of performance and just assume the M3 Ultra will perform with double the performance of the M3 Max in LLMs and that's it. There's no other special sauce to it. So the M3 Ultra (full die) will probably be about 60% faster than M4 Max and since the M4 Max is about equal to the M1 Ultra, the M3 Ultra will be about 60% faster than the M1 Ultra too. It will likely be great for large MoE models and a brilliant all-in-one solution but it's not going to rock the boat in terms of speed compared to other more complicated solutions.

[-]

Durian881@reddit

I'll just wait for benchmarks on actual machines.