Should I get a 14 inch M4 Max 128GB for 123B models?

[-]

Legcor@reddit

I have the 16 Inch M4 Max and I get around 4.1 tokens per second with first generation and 4.25 tokens per second with second generation, with flash attention and 2k context with your model Behemoth-123B-v2f-Q3\_K\_M. The backend is Koboldcpp. For a context of 16k the generation speed drops to 3.2 tokens per second on the first generation and 3.69 on the second generation which is kind of average typing speed level. The prompt processing took 400 seconds. The fans kick in but they dont get too loud. It doesnt bother me at all. Its less noisy on battery mode.

Reply

[-]

TheLocalDrummer@reddit (OP)

Thank you for the detailed report, comrade. Q3KM is not bad, but the speed is worrying. In your opinion, is it worth it? >The fans kick in but they dont get too loud. That's the 16 inch for you. A 14 inch will need to work twice as hard from what I've read. How's the heat?

Reply

[-]

prtt@reddit

> A 14 inch will need to work twice as hard from what I've read. Where did you read about this? I have heard no reports of throttling.

Reply

[-]

CommercialOpening599@reddit

Smaller laptop = Smaller fans = less ventilation

Reply

[-]

prtt@reddit

sure — still, no proof of throttling.

Reply

[-]

Legcor@reddit

No problem. I get around 60C degrees for short generations and never got more than 72C degrees even for longer generation or prompt eval. For me the Macbook was totally worth it. It's a tax write off for me, so it was a no brainer, especially if you consider the resale value and so on. I usually run 70b models where the speed is much better.

Reply

[-]

Difficult-Olive-1746@reddit

out of curiosity, did you end up buying a 14" or 16" m4 max?

Reply

[-]

TheLocalDrummer@reddit (OP)

I haven't bought one yet, but I'm probably going to get a 14" since it doesn't seem to make a huge difference in performance.

Reply

[-]

Difficult-Olive-1746@reddit

I see, good to know! I was thinking to learn more about LLM and getting an either 14" or 16" M4 Max. I really like the form factor and portability of the 14", but worried about the thermal and performance loss of the 14" after seeing threads like this https://x.com/ivanfioravanti/status/1856783964075352283. But looks like the performance loss is not that important in terms of local LLM, guess 14" should be good.

Reply

[-]

Its_not_a_tumor@reddit

I got mine a few days ago and have been testing. Honestly the 64GB would have been enough. You're just looking at the difference between if you want to run a 70B model really slowly with 8Q on the 128GB, vs a 4Q which runs much faster but can then run on a 64GB as well. I don't know of any 123B models that are really worth it.

Reply

[-]

GrehgyHils@reddit

How much RAM are the 4q 70b models using? I'm trying to figure out how close to the 64gb limit you're finding you are. As I might want to have a VM running as well as an llm

Reply

[-]

Its_not_a_tumor@reddit

35.3GB

Reply

[-]

TheLocalDrummer@reddit (OP)

I've got a 123B in mind that I know I want to run as a daily driver. From my experience, a Q4KM 123B will need around 90GB at 16K context.

Reply

[-]

tinny66666@reddit

The unified VRAM is amazing for the size of models it allows you to run, but it's nowhere near as fast as a 3090/4070. It's all a trade-off between speed/model size/cost so it depends how important those are to you.

Reply

[-]

TheLocalDrummer@reddit (OP)

I'd be very happy with 6 to 7 TPS given that's how fast generation is with an A40 GPU.

Reply

[-]

No-Refrigerator-1672@reddit

[M4 max is capable](https://ben.terhech.de/posts/2024-11-11-qwen-coder-25-32b-on-m4-max.html) of 13 tok/s with 32B Q8 model. Assuming the perfomance scales linearly and there are no bottlenecs, best case scenario you'll get 4 tok/s out of 123B model. You can use this rough estimation to make your decision.

Reply

[-]

ThenExtension9196@reddit

Thx for the breakdown

Reply

[-]

FreegheistOfficial@reddit

also need to leave room for ctx and prefix cache. 123B model @ fp16 needs over 100GB for 128k ctx.

Reply

[-]

SomeOddCodeGuy@reddit

I have an M2 Ultra 192GB Mac Studio, and I really don't recommend it for *that* model. I've seen the numbers; the M2 Ultra is still faster than the M4 Max (though I suspect the M4 Ultra will be a beast), and the 123b is on the edge of what my patience can take. You have to be *patient* if you make this purchase. In case you're curious, take a look at this post of numbers. Scroll down to Miqu-1-120b (roughly similar size) at 16k context to get an idea of times on the M2 Ultra. [https://www.reddit.com/r/LocalLLaMA/comments/1aw08ck/real\_world\_speeds\_on\_the\_mac\_koboldcpp\_context/](https://www.reddit.com/r/LocalLLaMA/comments/1aw08ck/real_world_speeds_on_the_mac_koboldcpp_context/)

Reply

[-]

Such_Advantage_6949@reddit

I have both m4 max and a rig of 1x4090 and 3x3090. And even on the rig with nvidia gpus and running with tensor parallel, i still dont really like the speed (about 16-19tok/s) let alone running on the max. You own the ultra and always speaks of the truth but seems like people dont really listen. People always say they are fine with whatever tokens/s but until they load up such big model, i am pretty sure most of them will just go and use claude/gpt instead.

Reply

[-]

_hypochonder_@reddit

I use 123b-iq3-xxs models and get 7tok/s and with more context I get 3tok/s. It's for me usable for rp. But if I want to e. g. analyse/generate code, you are right that you want more speed. I prefer 123b over 70b. But there is always the next new model waiting around the next corner which can change my mind :3

Reply

[-]

heartbleed_hack@reddit

I have a m4 max with 128gb topped out and we all just switched to using windsurf ide for code in actual projects with the new cascade feature - imo way better than local or using ChatGPT or Claude directly (windsurf uses Claude behind the hood and some proprietary models). For $0-$25 mo it’s the best purchase we have made, our new workflow is amazing

Reply

[-]

TheLocalDrummer@reddit (OP)

> It's for me usable for rp. Thanks hypo, exactly what I needed to hear.

Reply

[-]

Such_Advantage_6949@reddit

Yes u r right that use cases are important. For coding most of the time it is an iterating process, so it involves paste the code in and retry etc. Other than that bigger is always better

Reply

[-]

dammitbubbles@reddit

I went with 64gb m4 max because it seemed like from users benchmarks any of the 70b + models ran at 5tk/s which sounds unbearable to me.

Reply

[-]

Daemonix00@reddit

yes! but long runs will kill the battery. Even on wall power it needs more power from battery every time you run LLMs (big LLMs I mean, not just a 7B).

Reply

[-]

Musenik@reddit

My M2max 96GB MBP runs Mistral Large 2 (also Behemoth) q4 at 3+ t/s. at 32k of context it's between 1 and 2 t/s. These ML2 models are much smarter at larger quants, than people give credit to. They focus on the percentage of loss between quants, but that percentage, seemingly small, adds up. I remember when the Apple Newton had 95% handwriting recognition. That was true, but everyone hated all the mistakes it made, 1 in 20 letters! The fun thing is, I use my 45 watt charger meant for my Steam Deck! The downtime between generations (RPing) while I'm writing my text, is enough to keep the charge about the same.

Reply

[-]

TheLocalDrummer@reddit (OP)

M2 Max at 3 t/s??? Do you use Kobold? How long is the downtime usually? I usually stop mid-way and regen if I don't like where the gen is going.

Reply

[-]

Musenik@reddit

Good question! I've been afraid to look! Hey, the OS says the battery is at 99% health. Not bad for a year of majorly stressing it. I should note that I use Al Dente free, to limit charging to 70% of capacity. I'd heard that drawing power from a half charged lithium battery is the least damaging. Maybe so?

Reply

[-]

Daemonix00@reddit

Hehe i do use the steam deck power supply too! I never take more than 65watt with me travelling

Reply

[-]

fhuxy@reddit

These are tools, not investment pieces. Anyone seriously considering one of these for LLM is already planning to upgrade to the M6 Max, maybe even the M5.

Reply

[-]

lxgrf@reddit

I’d be fascinated to know how you think Space Black might matter

Reply

[-]

TheLocalDrummer@reddit (OP)

https://preview.redd.it/a36327k9at3e1.png?width=700&format=png&auto=webp&s=57eb5d19be95648d9cf253b70c7c32334817db3c

Reply

[-]

FullstackSensei@reddit

Two questions: do you absolutely need the portability? And why 123B, and not a (much) smaller model? If you're basing your purchase decision on the current state of LLMs, you might find that in 6-9 months 30B model will outperform current 100+B models. LLMs are still improving almost exponentially every 6-9 months.

Reply

[-]

TheLocalDrummer@reddit (OP)

> do you absolutely need the portability? It's a laptop first, LLM machine second. I'm upgrading from my M1 since I really want the speed for productivity & gaming. > If you're basing your purchase decision on the current state of LLMs, you might find that in 6-9 months 30B model will outperform current 100+B models. Yeah, but you could also say that in 6 - 9 months, a 100B model will be leagues ahead of current 100B models. Why limit myself to 30B then?

Reply

[-]

FullstackSensei@reddit

Maybe, but there's a trend towards smaller models as training pipelines evolve. They're cheaper to train and cheaper to deploy. The 100B+ Models also come with restrictions regarding use. The point I, alone with other comments, are making is: your money will get you much further if you reduce the specs of the Mac, and build an inference rig with GPUs. You can always VPN home using something like Tailscale to access it while on the move. You'll get more performance, possibly more VRAM, and can use it for really long sessions without worrying about heat, noise, or battery life. As always, it's all a tradeoff.

Reply

[-]

TheLocalDrummer@reddit (OP)

I understand. But spending thousands for a dedicated inference machine (that may require special wiring, special ventillation, special everything, special mobo, etc.) seems like a huge purchase for a lot of pain and worry. Meanwhile, an M4 128GB can be used for other things, can be carried around anywhere, is literally local, and can be used to play Stellaris. And I'd only have to add extra (admittedly double) on an inevitable purchase (the laptop).

Reply

[-]

FullstackSensei@reddit

Very valid point. Then maybe a Mac Studio with an M1 Ultra? LLMs are memory bandwidth bound, and the Ultra still has a lot more memory bandwidth than thr M4 pro. Separating inference from your work machine gives you a much better user experience. That's the thing I would do.

Reply

[-]

TheLocalDrummer@reddit (OP)

Yeah, that's way more interesting, I will consider that when I've got extra funds. Ultimately, the question here is whether I should splurge my planned laptop purchase and get a Largestral-capable upgrade.

Reply

[-]

FullstackSensei@reddit

To be frank, unless you have some privacy concerns or restrictions, or need to generate millions of tokens, it'll be way cheaper to get a chatgpt and Copilot subscriptions. In a couple of years, when the dust is settled, we'll know better about what hardware is needed for what.

Reply

[-]

isr_431@reddit

This is the drummer you're talking to, check his HF page to see the types of models he publishes. ChatGPT certainly isn't capable for that purpose

Reply

[-]

Such_Advantage_6949@reddit

I own m4 max 64gb and own both a rig of 4090 3090. And i would say, just go for it and buy the most expensive mac max. If u feel like it u can buy a rig with nvidia gpu later.

Reply

[-]

cheeseman_1000@reddit

I like your second point. It's good. I wouldn't have thought about it.

Reply

[-]

TheLocalDrummer@reddit (OP)

Big number good always

Reply

[-]

Valuable-Run2129@reddit

Currently the best open source model is a 32B model. Not 70, not 100, 123 or 180… the current smartest model is 32B.

Reply

[-]

Psychedelic_Traveler@reddit

What’s the use case for 123B models ?

Reply

[-]

Final-Rush759@reddit

Go with 16 inch which has 140W charger to meet the power requirement.

Reply

[-]

TheLocalDrummer@reddit (OP)

I've never had a 16 inch before.

Reply

[-]

matadorius@reddit

I think the reviews I saw online it is way too slow gonna wait a little bit to trigger the button tho

Reply

[-]

AIGuy3000@reddit

My M3 Max 40c 128gig gets about 4-5 tokens/s w/ Mistral Large. Qwen coder 32b gets around 20 tok/swith under 4k tokens, and drops to about 15 tok/s w/ 16k+ tokens which is useable IMO. Don’t expect to get more than 6-7 tok/s on 70b+ models there just too slow. At that speed I’m better off going to stack overflow for coding tasks.

Reply

[-]

ortegaalfredo@reddit

I think you will get 3 or 4 tok/s, I mean its faster than a PC but for 120B models I think you need a external GPU, 4 tok/s is torture.

Reply

[-]

me1000@reddit

I dont find the fans to be very loud, but 123B is going to be slow on the M4 Max (the size of the screen won't matter much). I haven't personally run anything that large, but I'd estimate around 5 tokens per second, max. Maybe slower? 72B a Q4 (MLX) runs about 10 t/s and after the first 500ish tokens we're talking 9 tokens per second. I already find that a bit too slow.

Reply

Reply to Post

52 Comments