To be up front, I haven't really done much with this, especially when it comes to managing multiple long contexts, so maybe there's something I'm overlooking.

Is it feasible to run 670B locally in that budget?

Without knowing the quantization level and expected performance it's hard to say. For low enough expectations, yes. Let's say you want to run the FP8 model at 10t/s per user so 1000t/s (though you probably want more like 2000t/s peak to get each user ~10t/s on the mid-size context lengths). That might not be possible.

Note that while 1000t/s might look crazy you can batch inference, meaning process multiple tokens at once, for each user. Because inference is mostly memory bound, if you have extra compute you can access the weights once and use them for multiple calculations. Running Qwen3-30B as an example:

You can see my 4090 'only' gets ~170t/s when processing one context, but gets 1335t/s processing 64 contexts simultaneously. That's only 20t/s per user and dramatically slower than the 170t/s because this is an MoE like Deepseek. For a single context only 3B parameters are used, but across 64 context nearly all 30B get used. For reference Qwen3-32B also gets about 10t/s @ batch=64 but only 40t/s @ batch=1.

Can Apple Silicon handle this effectively — and if so, which exact machines should we buy within $40K–$80K?

I think the only real option would be the Mac Studio 512GB. It runs the Q4 model at ~22t/s, however that is peak for one context (i.e. not batched). A bit of Googling didn't come up with performance tests for batched execution of 671B on the Mac Studio, but they seem pretty compute bound and coupled with the MoE scaling problems as mentioned before I suspect they will probably cap around 60t/s for maybe batch=4. So if you buy 8 for $80k you'll come up pretty short. Unless you're okay running @ Q4 and ~5/ts.

If someone has batched benchmarks though I'd love to see.

How would a setup like this handle long-context windows (e.g. 128K) in practice?

Due to Deepseek's MLA 128k context is actually pretty cheap, relative to its size. 128k needs 16.6GB so times 100 is a lot of VRAM. But, 128k context is also a lot. It is a full novel, if not more. You should consider how important that is and/or how simultaneous your users are.

What’s the largest model realistically deployable with decent latency at 100-user scale?

Are there alternative model/infra combos we should be considering?

It's hard to say without understanding your application. However the 70B range of dense models might be worth a look, or Qwen3. However, definitely watch the context size for those - I think Llama3.3-70B needs 41GB for 128k!

Qwen3-32B model might be a decent option. If you quantize the context to q8 and limit length to 32k you only need 4.3GB which would let run the you serve ~74 users and the model at q8 at very roughly 20t/s from 4x RTX Pro 6000 Blackwell for a total cost of ~$50k. Maybe that's ok?

I guess just to guess about Deepseek... If you get 8x Pro6000 and run the model at q4, that leaves 484GB for context so 30 users at 128k. Speed? Hard to even speculate... Max in theory (based on bandwidth vs size of weights supposing all would be active) would be 35t/s, though, so >10t/s seems reasonable at moderate context size. Of course, 8x Pro6000 is just a touch under $80k already so you won't likely be able to make a decent system without going over budget.

P.S. This got long enough, but you could also look into speculative decoding. It's good for a moderate speed boost but I wouldn't count on it being more than a nice-to-have. Like it might go from 10->14 but not 10->20 t/s.