Can 2 RTX 6000 Pros (2X98GB vram) rival Sonnet 4 or Opus 4?

[-]

x54675788@reddit

Easy and scientific test: go to lmarena.ai and test some models that you plan to use, then compare answers with Opus 4 or whatever. Ask another llm to compare the answers. Most models suck. Closest to Opus 4 is Deepseek which requires about 750gb of memory (but doesn't have to be GPU memory). If you go this route, go for 8-12 channel memory motherboards. The answer will be slower, but more accurate since the model is 671b parameter in size. You can run quantized variants and get by with less memory but quality takes a hit. Even deepseek, if you compare answers throughly, is slightly worse than your usual ChatGPT or Claude. Anything smaller than 671b right now is even worse. People hate to admit this, especially in this sub, but if someone swears their 30b model is the same as OpenAI in quality, they are being delusional at best

Reply

[-]

devshore@reddit (OP)

maybe the performance can be good enough in similarity if you prompt it better? Maybe the people that are reporting success have learned how to spoon-feed it, whereas with Claude, you can be lazy and not spoon feed or even give it any specific context when you send it off on a task?

Reply

[-]

x54675788@reddit

The thing is, unless what you are doing is error tolerant (writing a story, but even there you might slip in incosistencies), if you are doing coding or math, the tiniest difference in quality can throw off the entire result. The difference between a program that runs and a program that doesn't run. You can spoon feed a shitty model all you want, but it will stay a shitty model.

Reply

[-]

Monkey_1505@reddit

Cue everyone assuming your use case is the same as theirs, and giving strong, probably wrong opinions because of it.

Reply

[-]

devshore@reddit (OP)

I guess I shouldve specified "full stack web dev". I assumed "claude" gave it away, but forgot there are other dev languages.

Reply

[-]

Monkey_1505@reddit

Yeah, a use case would have been real helpful to mention, as the tendency is for every person replying to assume the OP's use case is the same as theirs (or even that their use case is the only use case that matters)

Reply

[-]

AdventurousSwim1312@reddit

The calcul is not right, think that 200€ is roughly 3M tokens generated with Opus. With that same budget, you can perform around 1000 hours of inference (assuming 30c / kWh energy) on that setup, and Qwen 3 235b should run at around 200 t/s batch 1 / 6000 t/s batch 64 (approximated from my tests on 1 rtx 6000 pro). So you can generate around 20M tokens for the same budget, with no artificial rate limiting (hence easier inference paralellism. There are actually 5 cases where this is really useful against api: - confidentiality, I got clients how have sensitive code, so being able to code with my own ai is a peace of mind for them, no shadow it - batch process : if you got large amount of text to label / tag / clean / translate, using Api will be a pain in the ass, with vllm the queue is auto handled - large scale multi agent : I'm experimenting with multi threaded agents, what can spawn up to 512 sub processes running asynchronously and waiting for each other. I have yet to find a provider that can sustain that kind of demand - training : this is blazing fast on that card, about 9x faster than on 3090 when I'm using fp8 training - image génération : in 512 images per second, flux will run in less than 0.5 second, and wan 2.2 2*14b will create 5s video in under 30s

Reply

[-]

power97992@reddit

$200 is for claude max , you can get > 100 million tokens with it…

Reply

[-]

AdventurousSwim1312@reddit

Mmm opus is 75$/ M token, so 200€ will bring you around 3M tokens. Sonnet is 15$ / M tokens, so about 13 M token with that budget. Not really the same usage tho, Sonnet and opus are designed for edge cases where you require a really good model, for most use cases you can go for a much smaller model and still get the value.

Reply

[-]

power97992@reddit

That is correct if you use the api but if you pay for a subscription, you get discounts on opus …

Reply

[-]

devshore@reddit (OP)

the elephant in the room is that claude is taking losses on dev clients on these plans, and so they will either further nerf the plan, or will have to start charging double or more.

Reply

[-]

AdventurousSwim1312@reddit

Maybe, but on an other hand, I can fill most of my needs with free apis and open source deployments. In the rare cases where I need to rely on paid LLM, it is when none of the free alternative can provide a correct answer, and the funny stuff is that most of the time, closed alternative also fail in this cases. Gemini hallucinate itself to death, and Claude outputs confidently an over engineered wrong answer, so I really see no point at the moment to pay a high price to anthropic

Reply

[-]

power97992@reddit

True, GPt 5 thinking is really good, I don't even use sonnet anymore and API Opus is too expensive

Reply

[-]

Karyo_Ten@reddit

>batch process : if you got large amount of text to label / tag / clean / translate, using Api will be a pain in the ass, with vllm the queue is auto handled OpenAI has a batch API, at half cost, results within 24hours. >image génération : in 512 images per second, flux will run in less than 0.5 second, and wan 2.2 2*14b will create 5s video in under 30s You'll have to link your workflow, on a 5090 with Flux-dev fp16 from last year, it took 45s per image or so, unless the new Flux is way faster, hardware doesn't explain the boost (only 10% perf diff with RTX Pro 6000)

Reply

[-]

AdventurousSwim1312@reddit

Check nunchaku, il using the default workflow in 512 * 512, in NVFP4, 12 steps + teacache, blazing fast https://github.com/nunchaku-tech/nunchaku

Reply

[-]

Southern-Chain-6485@reddit

Is it worth it to use nunchaku with such a hardware?

Reply

[-]

AdventurousSwim1312@reddit

Yep, basically Blackwell have specific optimisation for FP4 compute, so it is in this format that you'll get the most compute power (about 3500 TOPs) against a mère 200 TFOPs in native bfloat16 inference, and nunchaku is the only tool I found that already supports native FP4 (NVFP4 is still very new and have low support in the ai ecosystem, both LLM and computer Vision)

Reply

[-]

Southern-Chain-6485@reddit

Ok, but at such speeds, you may as well get more quality with fp8, or even bf16. There is no need to output images as if they are getting fired out of a machine gun, I think

Reply

[-]

AdventurousSwim1312@reddit

Yeah, I agree, its just that I just receive the card and I wanted to know how far I could push it ;)

Reply

[-]

Yes_but_I_think@reddit

You need 8xB200 to rival Sonnet 4. Run Deepseek 3.1 in thinking mode with 128k context at home in full fp8 precision.

Reply

[-]

devshore@reddit (OP)

So theu are renting out 500k machines to people for 8 hours a day for $200 a month? Lol they are about to rug pull (if thats the term) a bunch of devs. They must be losing 2k a month on each dev user

Reply

[-]

kmouratidis@reddit

Those systems can do insane batching. Not the single-user 30 t/s a random dude on a Mac gets, but 15000 t/s across hundreds of users. Let's say 30 t/s per user, so 500 concurrent requests, not all them from the same people or 24/7 (you need sleep, no?), so maybe a conservative 1500 users. Electricity cost of that machine is maybe $500/month, probably less. Cost to buy $500k? That's all per system. 8 of these machines in a rack, 10 racks in a row, 10 employees earning $120k on average. 50M costs to build, 2-3M to maintain. The $20 subscription on 120k total users works out to $2.4M... per month. Now double your electricity costs, double your employees, halve your users, and you still make $15 on every subscription with a <4 year payoff period. That's my napkin math anyway. Feel free to give is better numbers.

Reply

[-]

power97992@reddit

But training costs billions?

Reply

[-]

kmouratidis@reddit

Inference providers don't train. Also it's a one-off cost that only costs that much for the very few companies that have enough users where scale helps. Google says ChatGPT has 20M paying subscribers (>400M / month), so ~20x my numbers. And they make money from the API too, and probably from Azure and everyone else hosting their models separately. I think most people are still stuck in 2022 when they think about AI economics.

Reply

[-]

power97992@reddit

I was saying a lot of the cost for primary ai providers(model creators) come from training…

Reply

[-]

devshore@reddit (OP)

since it is in millions of peoples (if not billions) to have a good open model, is there no open-source way to collaborate to create even better models? Something where millions of people can donate some hours of compute power with their own hardware? Its not like AI is a niche thing where there wouldnt be enough people.

Reply

[-]

ArmadilloTM@reddit

This guy maths. (And is absolutely correct)

Reply

[-]

Monkey_1505@reddit

AI is generally unprofitable when considering infra, training costs. It's obviously a financial bubble, so yes.

Reply

[-]

xcheezeplz@reddit

Claude does not appear to be a single dense model and if not it would allow a lot more tokens/concurrency with that hardware...so that hw req might be overstated for a single user. Also breakage. Anyone using 8 full hours of processing is getting rate limited or sent to a fast model. Your average dev does not have it pondering for 8 hours straight. Half the day is spent writing prompts and reviewing the code, playing with and testing the results and some manual refinement. The token use in cli is actually light for the amount of work it produces by offloading a lot of work on the host.

Reply

[-]

Freonr2@reddit

You don't need to pay $200/month. Buy API credits and use via API. $200 is quite a lot of API for a month. Two 96GB cards isn't enough to run the larger open weight models that might start to compete with models like Sonnet 4. If this is for coding, Qwen3 Coder 480B is probably the watermark for being somewhat competitive and you still can't run that on two $8500 GPUs. And $19k is a ridiculous amount of API calls even if you consider the next 3 years of work. Depending on what you're doing you have to ask what your time is worth at some point when each API call is maybe a dozen pennies. Again if this is for coding, spending an hour or two debugging a problem your scuffed local model caused, or having to revise it over and over with the AI agent 10 times isn't worth it vs. just paying $0.35 to pay for a better model via API. Do not go into debt to buy GPUs. GPUs are a luxury item. Pay cash or don't buy at all. If you want to run the bigger models you can buy an Epyc or Xeon 4+ workstation with 768-1TB ram for about equal or maybe less than two RTX 6000s depending on what route you go. It won't be terribly fast (8-15t/s?) but at least you'll run a somewhat competitive LLM. Even then spending $10k-20k for local is outrageous compared to how many API calls that would be. I don't mean to sound anti-local, but you need to consider what it is you're really trying to do. Local is great for certain things, and some models you can run on a single $600 GPU are completely fine for using in instruct mode and have them bang out very specific functions, use for analysis, image gens, whatever. But, you said vs Sonnet 4 or Opus 4, so I'm making some assumptions here and I think you'd end up being disappointed with what your $18k worth of GPUs can actually do vs those models.

Reply

[-]

devshore@reddit (OP)

I would agree wit you on financing vs buying in cash. The reason this is different is that the alternative is to "rent" Claude code for $200 (they will most certainly increase this soon) which is worse than financing your own GPU even if the GPU is only 90% as good as claude. The open models will also get better even if they are 4 months behind the paid versions. So, given that the existence of AI is forcing you to have to use AI, and so the default option is to throw hundreds into the wind anyway, financing your own hardware seems better even if buying cash is an even better option, its still better than throwing money to claude in perpetuity.

Reply

[-]

Monkey_1505@reddit

Eh, you probably could. IQ3\_XSS with a bit of selective layer specific offloading, provided the CPU and system ram was tight.

Reply

[-]

serendipity777321@reddit

Don't forget electricity costs

Reply

[-]

ortegaalfredo@reddit

If you use for single-user agentic coding, the GPUs are active less than 5% of the day, so the cost in electricity is less than the cost of heating food with a microwave.

Reply

[-]

serendipity777321@reddit

It's not profitable if you only use them 5% of the day

Reply

[-]

devshore@reddit (OP)

I plan on renting it out at least overnight every day

Reply

[-]

kmouratidis@reddit

With my expensive electricity: * ~$0.05 / hour when idle, or $0.01 for the GPUs and whatever for the system * ~$0.4-0.6 / hour when fully utilized with TP, depending on model (e g. Max-Q uses half the power). For an average 8 hour / day usage, that's $150/month. Oof. Plus the upfront cost at $20k or so. Ouch.

Reply

[-]

serendipity777321@reddit

Ouch indeed

Reply

[-]

Independent_Hour_301@reddit

Short answer: no it is not enough to rival Sonnet or Opus. But... I assume, based on the models you mention, you want to do some coding, further I assume you probably have automated coding in mind. Based on this here is a recommendation about a possible setup for you: With one RTX 6000 Pro already you can run gpt-oss 120B. Which is actually awesome when it comes to function calling, which is very important for auto coding. As the RTX 6000 Pro is a multi instance GPU, you could split the second card into 4x24 GB and then run smaller models on them. I just take gpt-oss 20b as an example, or qwen3 32b q4 or qwq32b q8. Then, you can write your own model proxy an then take Claude Code and change the Claude Base URL to your proxy. In your proxy you need to map then a call from Claude Code to Opus to your gpt-oss 120b (which Claude Code use as Orchestrator), and a call to Sonnet you need to map to your smaller models. Then I also highly recommend to use Claude Flow on top (I am using Hive mind, not Swarm, but of course you can test yourself). I promise you, you will get very good results. Probably not the same, but very good. But you can also see from what I wrote: just running a model on the GPUs is not enough. You need to put in some work, like writing a proxy (but Claude can do this for you) and depending on the model you select, you also need to find the right parameters for it to work decently. I mean like for example, that qwq32b is only giving good results with temperature=0.6 and top_p=0.95. stuff like that. But after this work, you will have a nice setup. The biggest issue that I see is the context size for your orchestrator model, because this will fill up quickly and cause issues. At work we found that GPT 4.1 actually performs best as orchestrator, as it has a huge context window and still has not so much decay in performance than other models. Also it has great function calling ability, although not as good as gpt-oss 120b. But as you want a fully local setup, gpt 4.1 is out of scope. My final recommendation before you buy any GPU: Get an Openrouter account, write your own proxy to map Claude Code model calls to other models. Use Claude Code and Claude Flow with your proxy and experiment with different models from Openrouter which would also run on your planned GPU setup. If you find a combonation that you are satisfied with, go for it :) Same goes of course without Claude Code. Use Openrouter to find models that would fit on your planned GPUs and test them against Opus and Sonnet for your purpose and see if you are satisfied or not. Have fun. :)

Reply

[-]

Chance_Value_Not@reddit

I assume GPT-OSS will run fine with some good configs on a 5090 even. Especially if you have good memory bandwidth on your motherboard

Reply

[-]

TumbleweedDeep825@reddit

Just use opencode instead of fucking with Claude Code?

Reply

[-]

Independent_Hour_301@reddit

Actually I recommended Claude Code because of Claude Flow. Claude Flow is really awesome. If there would be something like "OpenFlow" or so, then of course you can use whatever you like. However, if you would ask Claude itself to write a proxy for you, it would take maybe 5 min from prompt to having it live and you do not have to change anything in Claude Code itself. Just go to your terminal and run: export ANTHROPIC_BASE_URL=http://path/to/your/proxy export ANTHROPIC_AUTH_TOKEN="whatever" If you use windows, I think you need to modify these keys somewhere in the registry. But I am no windows user, maybe you can do actually also just the same in powershell, idk.

Reply

[-]

TumbleweedDeep825@reddit

I must be a low IQ simpleton. I've been "coding" all of my life and I still can't find a use for agents. I can only review/test code so fast. How do you use them productively?

Reply

[-]

dametsumari@reddit

Nope. Outcome is probably slower and definitely worse ( see eg https://aider.chat/docs/leaderboards/by-release-date.html ). Claude and Gemini are better than any open model currently and you do not have vram for Deepseek even with those cards. Qwen would work but not as well.

Reply

[-]

-dysangel-@reddit

That chart finishes at the end of 2024!!!! There have been some spectacular open models released this year. If he runs GLM 4.5 Air or gpt-oss-120b he'll get Sonnet level performance.

Reply

[-]

dametsumari@reddit

No he won’t. I have tried them all and while they are nice for some niche tasks both Claude ( non haiku ) and Gemini 2.5 pro blow them out of the water At least in my coding tasks. Your mileage may vary of course.

Reply

[-]

Karyo_Ten@reddit

Depends on the coding you do, apparently GLMs models are very strong at frontend even GLM4-30b. For Rust backend engineering though didn't find any model good enough for my need, even Gemini 🤷

Reply

[-]

Holiday_Purpose_3166@reddit

FYI Qwen3 30B A3B 2507 both Instruct and Coder do well on Rust. The Thinking model spends more tokens for same quality output as Instruct - steering faster with Instruct/Coder. OSS-GPT-20B dataset corpus excels in Rust, better than the Qwen3 30B variants above in my senior Rust benchmarks. However, biggest caveat is somewhat poor in tool calling (according to others) and only works in Cline if you use llamaccp with the grammar flag fix. OSS-GPT-120B is good but operates similar to Qwen3 30B, making the smaller version more efficient choice. Where the 120B is excellent in reasoning for edge cases. For the OP, using 2x Pro 6000 will get very good results with any of the choices above. I'd hit at 120B on the fact for having much larger knowledge and much more efficient token generation than Qwen3 235B in my experience. I can load both 20B and 120B at same time in my system with a single 5090 - where the 120B is loading the experts on the CPU and remaining in GPU, leaving room for 20B. OSS-GPT-20B native at full context, and 120B also native at 120k context, otherwise it pushes VRAM out the window.

Reply

[-]

JonasTecs@reddit

Which exports on 120B you load in cpu? Is not super slow?

Reply

[-]

Holiday_Purpose_3166@reddit

Depends what you need it for. Normally use it for deep reasoning or edge cases. Project designing, etc. Wouldn't normally use it as daily driver, however running on Llamaccp is bearable above 35 toks/secs with Cline for some coding. RTX 5090 with Ryzen 9 9950X +96GB DDR5 6000 Offload as many experts into the CPU. LM Studio: 25-30 toks/sec Force experts to CPU Mx GPU offload Full Context Flash Attention 16 Threads Llamaccp: 35-40 toks/sec --ngl 99 -t 16 -c 0 -n-cpu-moe 23 Ideally use gpt-oss 20B over 280 toks/sec or Qwen3 30B A3B 2507 series just around 200.

Reply

[-]

JonasTecs@reddit

So what you use for dailing codimg with cline?

Reply

[-]

Holiday_Purpose_3166@reddit

Qwen3 30B A3B 2507 Instruct or Coder. Generally at 120k context with UD-Q4_K_XL. Although I can go up to 250k it seems to bog down about 100k as its not truly on GPU, but most my work finishes optimally under 100k which I'm satisfied with. Trialing now GPT-OSS-20B via Llamaccp as the grammar fix allows it to interact with Cline.

Reply

[-]

dametsumari@reddit

True but he was keen about Claude family models. They mostly are good in coding and to some extent tool use. I do mostly go, python and swift. Claude family is best in swift, go and python are about same for Gemini and Claude.

Reply

[-]

Coldaine@reddit

No joke, the new Grok coder fast punches way above it's weight on rust. Opus 4.1 and Chat GPT for high level plan review, Grok coder for implementation.

Reply

[-]

Michaeli_Starky@reddit

He won't for sure for anything that requires more than 30-40kB of the context

Reply

[-]

Monkey_1505@reddit

Nothing operates at that context length without very noticeable performance degradation.

Reply

[-]

Michaeli_Starky@reddit

Of course. That's why multi-agent setups are absolutely required for any real work. Smaller models with low quantization on top of that are falling short very fast.

Reply

[-]

Monkey_1505@reddit

Well, whatever consider 'real work', that's probably not something everyone is after.

Reply

[-]

Michaeli_Starky@reddit

OP is talking about replacing $200 Claude Max. Sounds like real work with large contexts.

Reply

[-]

Monkey_1505@reddit

Or they are just wealthy, and paid for the highest tier.

Reply

[-]

vibjelo@reddit

GPT-OSS-120B with native precision + full context length takes something like 66GB of VRAM, so enough with just one Pro 6000 for that :) GLM 4.5 Air ends up a bit trickier tho...

Reply

[-]

Resident-Dust6718@reddit

Yes, that’s what I was saying!!!

Reply

[-]

Super_Sierra@reddit

Benchmarks don't really matter, real world performance of your actual task does, and corpo dominates, and for most users, open source will do, unless you want the absolutely best experience. Complex tasks? Tasks that require intuition? Implicit understanding? High abstraction? Open source right now are pretty terrible at this, and have difficulties understanding anything that isn't literal, in your face. Look at all the tricks roleplayers and writers use to get their character cards to work in smaller models ( AliCat, author zero and other tricks ) for verification. Privacy? Open source obviously, but openrouter isn't the worst thing on the planet, and you can anonymize your data for higher prices. Open source will get better, but in my personal opinion, it hasn't even caught up to Claude 2 yet in most tasks outside of benchmarkmaxxing.

Reply

[-]

GravitasIsOverrated@reddit

Can you elaborate on the tricks RPers are using? I’m not in that space but I’m curious how you’d squeeze/fake more understanding out of the models.

Reply

[-]

xoexohexox@reddit

Check out r/sillytavern and you'll learn all about it. Context engineering, vector storage/RAG, fiddling with the samplers and chat/instruct templates, there's a ton of fun stuff you can do

Reply

[-]

Famous-Recognition62@reddit

Just followed that link and apparently it’s been banned from Reddit because it wasn’t moderated. That’s a shame as it sounded like the sort of community I wanted to join and learn from.

Reply

[-]

ninjaeon@reddit

r/sillytavernai

Reply

[-]

Famous-Recognition62@reddit

You beautiful creature!

Reply

[-]

Ok_Still_4308@reddit

That seems like a weird one to ban.

Reply

[-]

xoexohexox@reddit

Whoa I had no idea I read that sub every day

Reply

[-]

epyctime@reddit

r/SillyTavernAI maybe?

Reply

[-]

xoexohexox@reddit

Oh yeah that's the one

Reply

[-]

xoexohexox@reddit

Well check the sillytavern discord then I guess

Reply

[-]

Monkey_1505@reddit

All models are stupid tbh. SOTA proprietary models are just slightly subtly less stupid.

Reply

[-]

nomorebuttsplz@reddit

when you say "open source" are you talking about models under a certain size? Are you excluding deepseek, kimi, even qwen coder?

Reply

[-]

Super_Sierra@reddit

If you need corpo hardware to run a thing, it is essentually corpo. I love Kimi K2, and when I am low on openrouter money, I switch to that from Claude. Deepseek sometimes gets what I want but it has some personality issues. Qwen models objectively write like shit and are autistic due to their heavily finetuning.

Reply

[-]

Resident-Dust6718@reddit

Some of the deep seek R1 distilled models are actually shockingly good comparable to GPT three or GPT four

Reply

[-]

dametsumari@reddit

Yes. At work I use only corpo models for that reason. For hobby stuff I use mixture of self hosted ( studio with 96 gb ) and free tier Gemini currently.

Reply

[-]

Monkey_1505@reddit

That's a benchmark for coding. Most people using LLMs don't care about coding.

Reply

[-]

dametsumari@reddit

Combination of asking about Claude specifically and considering 2 RTX 6000 pros does make OP not part of most people.

Reply

[-]

Monkey_1505@reddit

Sure, but it also doesn't mean he is primarily interested in coding as a use case.

Reply

[-]

awebb78@reddit

Dude, you are pointing to a benchmark that hasn't been updated since 2024. In fact their new benchmark hasn't even been updated in quite a while. In order for a leaderboard to be informative it must be kept up to date. I was so excited when I saw Aider creating a leaderboard like this and I'm hoping someday it gets updated with the newer models like Qwen3 Coder (480B). Sure, their primary leaderboard says it was updated in August but the data is exactly the same with the same models as it was earlier this year. I'm not convinced that leaderboard is actually being updated.

Reply

[-]

inevitabledeath3@reddit

This isn't up to date and doesn't include models like DeepSeek V3.1, GPT 5, or Qwen 3.

Reply

[-]

dametsumari@reddit

In the link on the page there is the main list which is up to date : https://aider.chat/docs/leaderboards/

Reply

[-]

inevitabledeath3@reddit

Yeah I checked that list. No GPT-5, no DeepSeek V3.1, and no GLM models of any kind. They do have GPT-OSS and Qwen 3 for some reason. It's weird.

Reply

[-]

Down_The_Rabbithole@reddit

Q4 with QAT would potentially come close within the VRAM requirement.

Reply

[-]

Content_Cup_8432@reddit

Did you use Opus lately?

Reply

[-]

dametsumari@reddit

Yes. Mostly sonnet and haiku though.

Reply

[-]

TedHoliday@reddit

I am fairly certain the models they submit to Aider are not the ones running for customers. I have no evidence of this though.

Reply

[-]

Hoodfu@reddit

With an m3 ultra mac with 512 gigs, I can run deepseek v3/3.1 at q4 at about 380 gigabytes. Similar for GLM 4.5. These would probably be the closest I'd hesitantly say are in the same ballpark as one of the Claude Sonnets. With only 2x96 gigs of vram, you can't even run one of these at half size.

Reply

[-]

No_Shape_3423@reddit

What kind of speed do you get with DS and GLM? I'm seriously considering your setup. I can run Qwen3 235b Q4 on my 4x3090 at barely tolerable speeds, but the quality is not in the ballpark of larger models.

Reply

[-]

chisleu@reddit

The largest model I use on a Mac Studio 512GB is Qwen 3 Coder 480b 4bit which is 270GB. Active weight/parameter/neuron count is the biggest indicator of speed. smaller models have much higher token throughput and generally have better performance. 25 tokens per second with a simple prompt "write a two page report on the benefits of python"

Reply

[-]

GCoderDCoder@reddit

Just piggy backing here thatMac Studio is the way to go for personal LLM usage. I have the 256gb Mac Studio since I couldnt afford the 512gb at the time and I run Qwen3 235b at q4 and like it. I'm going to try higher quants since I have space. But I also use smaller models like qwen3 coder 30b or Gemma3 27b as smaller executors that actually write my code with certain workflows while the bigger Qwen3 235b is the planner that uses the logic needed for a good product. The big model can do it all. The smaller models can still be useful. Workflows and agentic integrations make the experience for something like chat gpt so much better than just using an llm in a chat window or on command line at home. I plan to buy a second 256gb eventually for mlx distributed to shard larger models across multiple instances so I can do Qwen3 coder 480b. OP sounds like he can afford the 512gb now but splitting between large state of the art models and medium code execution has been very fruitful for me. Now I'm working on the internet access and context management beyond cline. With persistence I still won't have all the features of chat gpt but I will have enough features for how I use these tools to rival what I would use a subscription for IMO. Owning it means constant ability to iterate without disruption and worries of them inevitably raising pricing. I also have a threadripper with 4 smaller GPUs24-32gb. I wish I had just gotten a 512gb Mac studio. 96gb won't go as far as 256gb could which also cant do what a 512gb Mac Studio could. Quantization helps fit larger models into smaller compute but there's a degradation curve as quantization gets higher. Everything doesn't need a huge model but spending this money I think you want big model performance not just doing research or serving an app to few users with medium llm capabilities on the backend I'm assuming. I take time pointing out that feeling the need to use Nvidia for larger models is self limiting to anyone who can't have their own datacenter. Mac Studio is best value for this use case in my opinion. Imagine multiple h100s with nvlink but slower yet very usable speeds. That is Mac Studio. https://preview.redd.it/5mfd89mek5mf1.png?width=4092&format=png&auto=webp&s=81910d54342a63d94e0ecf4b8bc64e17a0c6836b

Reply

[-]

EcstaticPut796@reddit

Please describe which tools you used to enable “Workflow and Agentic Integration.” Launching the LLM models via Ollama is straightforward, what happens after that?

Reply

[-]

nomorebuttsplz@reddit

Let us know if your 2 node system can split compute across gpus to increase PP speed!

Reply

[-]

chisleu@reddit

Be aware that throughput DROPS when you scale out with mlx.distributed, even when using thunderbolt interconnects. That is the thing keeping me from buying more Mac Studios. I'm keeping the mac studio to run really large models, and I want to build a blackwell for smaller models like Qwen 3 Coder 30b. Not that I NEED it with my macbook pro 128GB. Just that I believe it is a good investment because I believe more complex, reasoning models that fit in 96GB are going to start winning the open weight wars.

Reply

[-]

GCoderDCoder@reddit

Thanks. I appreciate the insights. I think this video below covered that a bit but I think it's much more usable than me running something like qwen3 coder 480b at a functional quant on system ram. I'm aiming for the same goals as you sounds like. I agree they are improving the efficacy of models at smaller sizes but I think about what's the minimum I need now vs how to be flexible for where it could go. I also think the math on AI investment isn't adding up. Investors don't want to wait 10years for a return so something will be changing in the pricing I imagine. I appreciate the feedback. These are expensive mistakes to make! Ask me how I know... lol https://youtu.be/d8yS-2OyJhw

Reply

[-]

No_Shape_3423@reddit

Thanks! What happens when the context window fills will real work, something non-trivial?

Reply

[-]

chisleu@reddit

depends on the workload. If it is agentic interaction like coding agents (cline, roo code, etc) then you get the benefits of prompt caching in lm studio. token throughput stays really usable. If you have unique, HUGE contexts that you need to process then the mac is going to take a long time to process the context. I don't have an effective way to benchmark that, but that's not my use case.

Reply

[-]

Hoodfu@reddit

With DS I'm getting about 16 t/s with unsloth gguf, and around 19-20 with the ds v3 q4 mlx version. I don't have numbers on glm offhand.

Reply

[-]

AppearanceHeavy6724@reddit

prompt processing I bet is around 50t/s at most. unusable.

Reply

[-]

xxPoLyGLoTxx@reddit

I always love these comments because it's invariably an nvidia shill who has like 16gb vram and thinks because they can run a 12gb model at 300tps it makes them cool.

Reply

[-]

AppearanceHeavy6724@reddit

You can love whatever shit you've made up my man, but Apple is ass at prompt processing, even 3060 tears a new one in Apple at that parameter.

Reply

[-]

xxPoLyGLoTxx@reddit

Yes with your 3060 enjoy running your 8gb models. Very capable!!

Reply

[-]

AppearanceHeavy6724@reddit

Mang, we are talking about prompt processing speed. Which is worse on Apple across all sizes. Comprendo? If I needed cheap VRAM i'd rather buy Mi50.

Reply

[-]

xxPoLyGLoTxx@reddit

You keep on beating that drum. It's rather meaningless. Have you ever even used apple silicon? Probably not. Having high prompt processing with 8-12gb models is pointless. It just serves up the slop response faster - that's about it.

Reply

[-]

AppearanceHeavy6724@reddit

Mang you are making me tired. I have 2 cards and soon be runing 28 GiB vram. 30B-A3B coder, or Mistral Small or any other 12-32GiB model will fly on that hardware. 20+ model is all I need mostly. Anything less than 300 t/s is unsuable in my book for coding. 500-600 neede for comfortable use. It is you who is keeping beating this drum - if it is okay for you 50t/s PP fine, enjoy processing your tiny file 20 sec.

Reply

[-]

xxPoLyGLoTxx@reddit

Ah yes. Anything less than 300 t/s is unusable? What a delusional take. That's honestly the most ridiculous thing I've ever seen in this entire sub. And I've seen plenty! Honestly if that's how you really feel then I have no idea what else to say. I'm speechless...

Reply

[-]

AppearanceHeavy6724@reddit

> Anything less than 300 t/s is unusable? ...for coding. For ERPism and chatting even 10 t/s PP is fine. Look no one is reading our convo, no one will convince each other, we should stop.

Reply

[-]

xxPoLyGLoTxx@reddit

Im stopping but your tps reqs are absurd. Just as an example, I get around 75 tokens / second with gpt-oss-120b. A fully coded response appears within what? 10 seconds? The fact that such a speed is unusable to you is just crazy. It's like saying "I can't wait 10s - that's unusable. Only speeds of 3s or less work for me.". Just nuts. Wild view.

Reply

[-]

AppearanceHeavy6724@reddit

_PROMPT PROCESSING_ dude - do you know what that means?, not token generation. A single 500 lines source code file is 5 000 tokens. With 50 t/s at zero context you'll be at 20 t/s at 5k context (due to quadratic attention). So to get your file processed you have to wait at least 100 seconds. For 1000 lines file you need to wait 300 seconds, at least. RAG is even worse, as the code might be partially saved in cache, with rag it is always different documents in context. I do not know what you code with 75 t/s prompt processing. Code foe ants? Oneliners?

Reply

[-]

xxPoLyGLoTxx@reddit

No the 75 tps is token generation - not prompt processing. I have no idea what the prompt processing is because I don't care. Usually it's around 3-4x token generation speeds. It's not returned by default in lm studio or open-webui server. If you think a Mac has 50 tps Prompt Processing for a model that fits into vram then you are even more ridiculous than I originally thought. Where did you even get that number? Just pull it out of your ass?

Reply

[-]

AppearanceHeavy6724@reddit

> I have no idea what the prompt processing is because I don't care. Why the fuck I should talk with such an inept person that does not even know what they arguing about? > Just pull it out of your ass? Pull out of your ass idiot. Conversation by GP was not about oss-120 even a worst potato will run fast, it was about deepseek 3.1 at Q4.

Reply

[-]

xxPoLyGLoTxx@reddit

Lol you are running deepseek 3.1 on your 3060? Riiiiight. PP for me, when I checked, was always at least 150 tps+. I have no doubt it would be at least 200-300 tps on most MoE models. Tell me how to check and I'll gladly do it in LM STUDIO or open-webui. But you are just making up numbers. You don't even have 28gb vram and you are talking about PP for deepseek 3.1 which you couldn't run on 8x 3060s. Again I dunno what you are smoking. Weird double standards that's for sure. You make up 50 tps PP, mock it, and at the same time, you can't even run the same model PERIOD.

Reply

[-]

AppearanceHeavy6724@reddit

We agreed to stop. I think this is the right place to.

Reply

[-]

Hoodfu@reddit

Hah I'm using it for text to image and text to video prompts en mass with scripts etc behind a big interface I've built and it starts spitting out response tokens to a \~300-400 token input prompt in less than a second, but go ahead and keep making stuff up to fit a narrative.

Reply

[-]

AppearanceHeavy6724@reddit

Who are you kidding, you pressed refresh. Of course you'll get 0.35 sec to first token then.

Reply

[-]

Hoodfu@reddit

You're sitting here trying to convince someone who uses this day in and day out that it's unusable.

Reply

[-]

AppearanceHeavy6724@reddit

Buddy I am not talking to you anymore as no one reading us. It has become our personal convo, not doing that.

Reply

[-]

Super_Sierra@reddit

Way more than that. I can still see you hate macbooks, did the ceo fuck your wife or something bro?

Reply

[-]

AppearanceHeavy6724@reddit

very edgy. Their CEO is gay fyi. Anyway I know they do not teach it in high school you go to, but Apple silicon is terrible at prompt processing.

Reply

[-]

Super_Sierra@reddit

Idk about all of that, but you reply to every damn comment about macbooks bro, its time to move on.

Reply

[-]

AppearanceHeavy6724@reddit

Do macbooks make you hard? Because the GP was not talking about macbooks, he mentioned M3 Ultra, which is not a book.

Reply

[-]

No_Shape_3423@reddit

Thank you. For single-user inference, it seems hard to choose 4 x RTX 6000 Pro ($30k) over an ultra 512 gb ($10k).

Reply

[-]

prusswan@reddit

You can just get a single Pro 6000 and make up the rest with RAM, should be faster in most situations with MoE models.

Reply

[-]

oh_my_right_leg@reddit

How would that work for processing and generation speeds?

Reply

[-]

prusswan@reddit

Anything that fully fits in VRAM will be fast. Larger models that go beyond but fit within RAM will be manageable.

Reply

[-]

Yes_but_I_think@reddit

Never a bad decision in today's scenario. For optimum quality you never go below q6_k ever. Context first, allocate full free space for your contexts, then for q6 of your model, then choose such a model size which can run this within VRAM fully. That's how you get a useful reliable system that can be hooked up with Roo or any agentic systems. Otherwise you will use it only for vibe check. Next, wait for fresher models that fit all these criteria and stay with your earlier one

Reply

[-]

prusswan@reddit

The crowd that buys a Pro 6000 can get RAM up to 1TB if they want to

Reply

[-]

chisleu@reddit

Right now, you are at a weird price point. The most effective thing you can buy is a mac studio. unfortunately, clustering mac studios isn't really effective so scaling the setup doesn't really work. However, I don't recommend you buy a mac studio. I recommend you buy a 128GB macbook pro and use that instead. Smaller models like Qwen 3 coder 30b a3b can handle intense use cases like Cline. Other models like GPT OSS are really good at translation. It's much satisfying getting 75 tok/sec out of a MBP... Nothing is going to be as powerful at sonnet yet, but open source models are trailing SOTA models by only 9 months.

Reply

[-]

Yes_but_I_think@reddit

It's not the thing to buy. You'll age before if does anything useful. Normal usage is 50k input and 1k output on a infinite loop till job is complete. Token generation speed matter way less than prompt processing speed.

Reply

[-]

chisleu@reddit

\> You'll age before if does anything useful Fucking what? \> Normal usage is 50k input and 1k output on a infinite loop till job is complete Fucking what? the most commonly automated use cases != "normal usage". \> Token generation speed matter way less than prompt processing speed latency and throughput are both important and a $12k mac studio vs a $15k single blackwell PC is a no brainer. The blackwell will outperform the mac on models that fit entirely in VRAM and be useless outside of that. The mac will be able to run much larger models in the 300GB range, but only models in the 32GB range are going to be fast enough for interactive agentic use.

Reply

[-]

Super_Sierra@reddit

Don't bother, they think 500$ electric bills are not signs of being a cuck.

Reply

[-]

No_Efficiency_1144@reddit

I still don’t understand why you and the other mac users are okay with waiting 10 minutes for every response, which is what you get if you run large models and contexts on the best macs. I don’t have a bias against Apple because I am typing from iPhone, but I cannot see how 10 minutes per response is okay.

Reply

[-]

chisleu@reddit

I'm sorry, but you don't know what you are talking about. The only use case where you are going to have minutes of waiting for first tokens if you load a huge model with a huge context and shove 200k tokens in for some reason. No one actually does that. Everyone is using some sort of agentic interface like Cline/Roo/Claude/Blah which builds context in chunks slowly over time. That said, I don't recommend buying anything larger than 128GB right now because \~32-64GB models are so good. For that use case, a single big GPU is the king of the ring right now, but a Macbook pro is a great alternative.

Reply

[-]

No_Efficiency_1144@reddit

Yes I agree with some of this. There is a wide variety of use cases for LLMs in different fields though. You’re assuming other people use the models the same way you do. In particular coming from a coding perspective. I am more interested in quantitative analysis, data processing, industrial control, strategic agentic decision making and automated financial trading systems, for example. For these areas reasoning over a context of 64-128k tokens with Minimax M1 80k or Deepseek 3.1 seems to still be best. I don’t currently have a big local rig but I think I am probably still in the “just stack GPUs” camp still.

Reply

[-]

chisleu@reddit

No, I'm not the one assuming things. I understand that there are lots of different use cases for LLMs. But for every automated use case that is well funded enough to tilt the charts on openrouter, there are hundreds of users using interfaces like Cline and Roo code.

Reply

[-]

No_Efficiency_1144@reddit

That is fair yeah a common purchase reason is just to use Cline/Roo style. A reason I focus on using LLMs more for math/quantitative problems is because LLMs are stronger there, relative to humans. Like LLM code is definitely worse than most post-grad human code, but LLM proof finding and quantitative analysis can be a lot better than most post-grad human attempts.

Reply

[-]

Willing_Landscape_61@reddit

"The most effective thing you can buy is a mac studio." Only if you don't care about prompt processing speed.

Reply

[-]

chisleu@reddit

ugh, the PC fanatics here are ick

Reply

[-]

Uninterested_Viewer@reddit

This has nothing to do with Macs.. it's about ANY of this class of machine with the lastest generations of shared/unified memory. Several companies are making them now, but Apple still has the fastest and most common and, therefore, is usually brought up for this class of computer. This has absolutely nothing to do with some juvenile "Mac vs PC" fanboyism.. just stop

Reply

[-]

Outrageous-Win-3244@reddit

I am using ktransformers with Kimi K2 1T, Deepseek v3.0 671b, and Qwen 3 coder 480b on the following system: - Nvidia RTX 6000 PRO 96gb, - AMD EPYC 9750, - 12x64 GB DDR5 5600 RAM. The KV-cache is in the GPU (Q4_K_M), the model weights are in the DRAM (Q8_0). It gives me 15/20 output tok/s. It can serve one user (me). My CPU has 256 threads, but only 128 is used by ktransformers for some reason. Didn't investigatem because the current speed is sufficient for me. I like the privacy and cost efficiency of my setup. My most used model is Kimi-K2 at the moment. I wouldn't switch to Claude 4 sonnet, I hope this helps. Gyula

Reply

[-]

alok_saurabh@reddit

How did you mix different quants for cache and weights? Where can I find a good tutorial or walkthrough for ktransformers ?

Reply

[-]

truth_is_power@reddit

2 RTX pros that you own are better than a 200$/month subscription that they will nerf secretly without you knowning. plus, you can demonstratively do more with your own GPU's than chatGPT/sonnet 4 etc. their image generation sucks balls. you can do better with 8gb and comfyui, for example. if you're smart enough to set the GPU's up, I would bet on yourself and not the mega corp.

Reply

[-]

power97992@reddit

Dude no way glm 4.5 air is better than gpt 5 or gemini 2.5 pro … glm 4.5 full is worse at certain coding tasks than gemini flash!

Reply

[-]

Monkey_1505@reddit

Try reading bro. This reply has nothing to do with what he said.

Reply

[-]

prusswan@reddit

Wan 2.2 default settings take more than 10 minutes for that 5s clip (closer to 30 minutes on lesser setups). But this is pretty much prosumer SOTA.

Reply

[-]

ShengrenR@reddit

How does their image generation do badly? Just looking at spec sheets I'd expect them to be \~2-3x a 3090 which does pretty decently - 'sucks balls' compared to what?

Reply

[-]

eidrag@reddit

I think they mean sonnet/opus image generation, not that gpu

Reply

[-]

ShengrenR@reddit

Ah yes, that reading makes sense. I was like.. do you know how big you could make a wan 2.2 image with two of those things?? Lol

Reply

[-]

Monkey_1505@reddit

Well look at it this way - maybe the models you can run here would be slightly lower in benchmarks. But you can also run image edit, txt to image, image to image, image to video, text to speech, speech to text, agents etc etc etc. The breadth of things you can run on your own hardware is larger than any individual AI service, and some of those models will actually be better than anything on such services.

Reply

[-]

Polysulfide-75@reddit

Those models take many hundreds of gigabytes of RAM.

Reply

[-]

Ok_Warning2146@reddit

Better one RTX 6000 Pro plus 1152gb RAM such that u can run the big MoE models while loading the KV cache to your GPU.

Reply

[-]

Pogsquog@reddit

[https://gosuevals.com/agents.html](https://gosuevals.com/agents.html) \- Kimi-k2 and Qwen3-Coder-480B-A35B-Instruct can give similar performance to sonnet 4, depending on the task. The main issue is that you need a system with a very large amount of fast ram (and lots of memory channels) to get acceptable performance with these models, as the models are huge - e.g. something like an EPYC with 12 channel high speed ram. A 4 bit quantized version of Qwen3-235B-A22B-Instruct-2507 would likely give good performance on the GPU, though not quite as good as Sonnet. You can put your system specs into hugging face and it will show you what will run on your system (on GPU or CPU, you can switch between them). https://preview.redd.it/ta24twk015mf1.png?width=2374&format=png&auto=webp&s=353a96b6d03661831a05cf0a9ec1f6b22df8fbbc

Reply

[-]

devshore@reddit (OP)

That says qwen3 is close to sonnet 4, but doesnt say what "quant" etc

Reply

[-]

SillyLilBear@reddit

Not even close

Reply

[-]

kor34l@reddit

qwen-3-coder (the fat one) is pretty damn close. like, *crazy* close.

Reply

[-]

oxygen_addiction@reddit

Not even that close to Sonnet, let alone Opus. Stop spreading lies. Glm4.5 is the only one in the ballpark and even that is a step behind.

Reply

[-]

kor34l@reddit

https://livebench.ai/#/?Coding=a&Agentic+Coding=a benchmarks aside, I use Claude and qwen 3 coder both, every day, for complex programming. I have only tested them extensively with C++ and Python, but they are very very close. Accusing me of spreading lies rather than merely being mistaken (though i don't think either is the case) leads me to wonder if you have some sort of bias or agenda or motive, or if you're just naturally hostile whenever you disagree with someone.

Reply

[-]

oxygen_addiction@reddit

Saying *“spreading lies”* was probably a bit melodramatic and over the top. I apologize for that. As for the matter at hand, I’m (unfortunately) a Claude Max user, simply because nothing else writes C++/Rust/Java as well as it does. I’ve tested just about everything out there via OpenRouter/Cerebras, directly from the companies, and through various frameworks (TaskMaster, RooCode, Kilocode, etc.). Claude Opus 4 still stands a step above anything else available. Qwen3 Coder 480B-A35B runs into the same issue as Gemini 2.5 Pro: it gets stuck in logical loops during agentic tasks. GLM4.5 doesn’t suffer from that problem and feels more consistent across the entire context window. Claude sometimes hits this as well, but it's smart enough to backtrack out of the loop and analyze the problem again or try something else (which is ocasionally a problem, because it gives up too easily). GLM4.5-Air, on the other hand, still gets tripped up by silly things, but overall seems a bit smarter than Qwen. GLM4.5 is the only thing that comes close to Sonnet 3.7-4 in my experience. That’s been my experience so far, after fairly extensive testing.

Reply

[-]

kor34l@reddit

Perhaps our usage methodology differs, as the problems I encounter with both are different than what you describe. I use both from the Gentoo Linux command line (bash) using Claude Code (with a wrapper to load Qwen vs claude)

Reply

[-]

oxygen_addiction@reddit

Intersting. What problems did you encounter with both of them?

Reply

[-]

kor34l@reddit

usual stuff, occasionally hallucinating class and object and sometimes even function names, ignoring certain guidelines given in the prompt, occasional bad assumptions, etc. I've gotten it pretty minimal by reading a lot of suggestions and seeing what works, and as long as i supervise it, the mistakes aren't too hard to avoid or correct.

Reply

[-]

emprahsFury@reddit

I'll be honest, when I see this sort of complaint, I just think you're mainly benefiting from the massive company paying prompt engineers to fix how you're using the model, not benefiting from the computer scientist PhD making better models

Reply

[-]

Final-Rush759@reddit

Testing is not enough. You have to use it for at least a project or two and learn how to interact with it. Qwen 3 coder is amazing with Qoder IDE (Qwen IDE). GLM4.5-Air is not good at all, making up things doesn't exist. Haven't use full version of GLM4.5. I think most of Claude's advantage is in their system prompt and agents, not their models. Others start to catch up on that. I think Claude still have some advantage.

Reply

[-]

TumbleweedDeep825@reddit

Agree. I hate anthropic and I hate cloud models. But for right now I'll pay the claude max tax because nothing else comes close.

Reply

[-]

robertotomas@reddit

Agreed. I use both qwen coder and gemini 2.5 pro heavily right now. I find qwen is overall like 90-95% as good for me, if we don’t count how many turns it takes to get there. Gemini feels smarter, and i do rely on it for my “bigger asks”, but frankly they both get things wrong at the same rate. If the work you have is highly repetitive and requires meticulous detail, qwen is actually better.

Reply

[-]

M3GaPrincess@reddit

Kimi k2 is rather good, I think ot beats glm4.5

Reply

[-]

SillyLilBear@reddit

Not close at all

Reply

[-]

power97992@reddit

Last time i used qwen 3 coder, it was a lot worse than gemini 2.5 pro or claude 4 sonnet

Reply

[-]

dr_manhattan_br@reddit

No, Claude 4 is at the same level or above Deepseek R1. As we don’t know exactly the size of Anthropic models as the company does not open source the model weights. We can presume they are similar in size of an R1 model. With that said, you need around 1TB of VRAM or a Nvidia server with 8x H200 141GB of VRAM to load the model, KV cache and get something similar of Claude 4. If you are thinking on quantized models, you save memory usage but trade performance and quality. So, you are basically running a less capable model. But those very LLM include information that you may not be using. Different languages included, specific domains of science, biology, history or even general knowledge from internet that you may not need. Comparing those comercial models and something that you can run locally is not a good way of comparing

Reply

[-]

faldore@reddit

You can run GLM 4.5 Air on that. It's no Sonnet - but it's quite capable.

Reply

[-]

Holiday_Purpose_3166@reddit

I have an RTX5090 and I'm pondering a RTX Pro 6000. Not to confuse with 6000 Pro Ada generation. Thanks nvidia for confusing names. I spend well over 50 millions tokens a day on different projects with large codebases and prototypes. Serving my own inference is far cheaper than renting via API. I can even use the Free models in OpenRouter for an occasional fart to test differences in very rare cases. That being said, depends on your use case. You can easily play with top open-source models more comfortably with 2x of those. Some models will definitely require some quantization but the small loss in quality is generally worth it than someone trying a Q1 of DeepSeek on a potato PC for 2 toks/sec and calling it a good day.

Reply

[-]

Resident-Dust6718@reddit

Also, I meant to say V ram not the ram.

Reply

[-]

Ok-Watercress3423@reddit

depending on the use-case, but yes GLM-4.5 beats the claudes on a couple benchmarks and not far behind in others. main issue will be tok/sec. there are many more uses cases for local, and with the patched P2P driver you might even consider 4x 5090s for more compute over 2x 6000s.

Reply

[-]

Commercial_Slip_3903@reddit

no not really. and remember that you’ll need to update your gpus too whereas the frontier models will just get better over time for (likely) the same or similar cost

Reply

[-]

tvmaly@reddit

What about using the API for the pro models only when needed? That would save quite a bit over $200 a month

Reply

[-]

getfitdotus@reddit

So I run glm4 air and it’s very good I want 4 of those to run the larger variant. I think this would be on par or even better and faster in some cases. I have 4 of ada6000s and air fp8 runs at 100tks.

Reply

[-]

BoJackHorseMan53@reddit

GLM-4 can

Reply

[-]

PaxUX@reddit

If you're a business spending this money then talk with your account to see the best way to write it off. The point of 300 a month is you get the latest hardware without any of the issues like maintenance and device failures. It's a convenience. Do it yourself is best for hobbyists where if broken for 4 days it doesn't loose you money. So it is about your uscase

Reply

[-]

waywardspooky@reddit

i think it really boils down to if you'll miss the money or not. if you can spend it and not miss or even have to think about it, ownership and the ability to run whatever you want has significant value, however you won't be able to rival closed models - from some post here recently i think it showed that opensource lags about 9 months behind closed models in terms of capability. so those are things to take into consideration. how much would you need to spend to be able to run cutting edge opensource models, is the tokens per second you'd be getting on that hardware acceptable, would you miss that money, and are you ok with the average 9 month lag between closed source and opensource catching up.

Reply

[-]

ortegaalfredo@reddit

You need a little more VRAM to run GLM 4.5, that in benchmarks is better than Sonnet4. But with that setup you could run Qwen3-235B that is very close, though.

Reply

[-]

-dysangel-@reddit

Test out GLM 4.5 Air, most people feel it's around Sonnet level in real world use. I'm not sure what kind of performance you'll get on those cards GPUs, but Air will load fully into VRAM even on one of them so it should be pretty good.

Reply

[-]

oxygen_addiction@reddit

Full glm4.5 is around Sonnet level. Air feels smarter than qwen 3 coder full though.

Reply

[-]

xxPoLyGLoTxx@reddit

You are shilling hard for sonnet in all your comments. I'm glad you dig it - I used it in the past and it was solid. But I've ceased using all corporate AI for months. No need with local models and how sophisticated they are nowadays. I primarily do web coding. I've found that gpt-oss-120b gets me what I need in a few prompts. If that fails I'll try some qwen3 or glm-air. Ironically, I've not been super impressed with glm-4.5 but have only used it limited times.

Reply

[-]

Karyo_Ten@reddit

70tok/s with vllm tensor parallelism and PCIe gen5 x8 link. (Asus ProArt Creator board)

Reply

[-]

Skystunt@reddit

No, not really. 192gb is nowhere near enough for top models, you should look at a mac or make a system with older tesla cards or data center, slower but lets you use better models. For example 10x mi50 would be around $1k but will give you 320gb vram Am older m1 ultra will give you 128gb for around 2k An m2 ultra with 192gb vram will get you the same capacity as 30k worth of cards but with way leas speed(still worth it) Don’t finance these cards, it’s not worth it, in coming years other cards with more vram will come out that will be even cheaper, the rtx6000 has a certain early adopter tax

Reply

[-]

emprahsFury@reddit

Recommending these cards that were shit when they were released 5 years ago and are barely hanging on with support form a tier 2 company is complete bullshit when this dude is talking about dropping almost $20k. And not just 1 of them but 10 of them. The MI50 is not just missing software support but hardware support. 192gb can run anything that has been released except llama3-405b, llama-maverick, deepseek, and kimi, and glm 4.5. That's 5 models across 5 years of development.

Reply

[-]

prusswan@reddit

Better than giving money to cloud services that also take your data. Recommending MI50 that even the Chinese is throwing out (that's why those are looking cheap)... let's just say there is a difference between cheap and being too cheap

Reply

[-]

Opteron67@reddit

comment are harsh. I made good use of qwen ccoder 2.5 32B

Reply

[-]

prusswan@reddit

I ran that on 8GB with sub-1 tps..now it is an easy three digit tps

Reply

[-]

kmouratidis@reddit

Yes, small models are great, I use Mistral 3.2 / Devstral small... but OP asked about 2x Pro 6000 and Claude 4. Your setup doesn't match the quality *or* the speed.

Reply

[-]

bick_nyers@reddit

3 of them can probably run Qwen 3 Coder.

Reply

[-]

Uninterested_Viewer@reddit

For very specific tasks and, if you're willing to put in a lot of work to optimize for that task.. then yeah, you can get close. Your money and time sink is going to be far deeper than "$300/mo vs $200/mo", though. To me, the biggest elephant in the room is that we're taking a brief moment in time to draw this comparison in a fast moving space whereas purchasing $20k in quickly depreciating hardware is a longer term commitment. Are you going to still be happy with "close to sonnet" performance 6-9 months from now when sonnet is considered a legacy LLM and is so far behind SOTA that nobody would even consider using it? Maybe local models will keep pace, but we have no idea about the hardware needed to run them. Sure, you can sell your kit at that point, but if you need to sell due to not being able to run the best models, that means the used market price will have also dropped considerably due to that same fact. Tldr; you'd be taking on far greater risk than your post implies. I just don't think trying to compete with SOTA models locally makes any sense outside of specific requirements of either security or very niche tasks that even SOTA generalists aren't great at. Or learning.. Tons more money, tons more time, and tons more risk for *maybe* comparable results for *some* tasks if you do it all right.

Reply

[-]

Final-Rush759@reddit

Qwen 3 coder is amazing for coding. You probably need at least 3-4 of RTX 6000 pros.

Reply

[-]

DbrDbr@reddit

That’s a good plan. But not today. I would wait for llama 5 or even 2026 or 2027 models. I m also considering this option but at other costs for result

Reply

[-]

Karyo_Ten@reddit

>I would wait for llama 5 or even 2026 or 2027 models. Is that a joke answer? Not even sure llama 5 is coming given the state of llama 4. And waiting has a huge opportunity cost, if OP uses $200/month of subscription, they get more than that in value from their AI, so they would still pay for it. And it's possible that hardware in 2027 gets even more costly or that there is a shortage.

Reply

[-]

DbrDbr@reddit

I m not up to date. I used ai to research. I really do consider myself this option. What would you propose. Send my on the right path pls

Reply

[-]

Karyo_Ten@reddit

>I m not up to date. I used ai to research. So blind leading the blind. >I really do consider myself this option. What would you propose. I don't know, do you make more than $200 from AI like OP?

Reply

[-]

dabois1207@reddit

Are you referring to renting the gpu’s on cloud or the purchase comes out to $300 a month?

Reply

[-]

Resident-Dust6718@reddit

Honestly, I think like two RTX pro 6000s with 96 GB of the ram would be far more than enough to outdo ChatGPT both in terms of LLMs and image generation AI

Reply

[-]

karolkt1@reddit

So why companies buy H200s and B200s? You will never come close to closed source + cloud in terms of pure quality

Reply

[-]

az226@reddit

No

Reply

[-]

zasura@reddit

You cant outdo opus with anything but you can get close to sonnet with deepseek or similar high quality open source models

Reply

[-]

Sicarius_The_First@reddit

**No**. not yet. once grok3 will be released as open weights, AND finetuned, then maybe...

Reply

[-]

devshore@reddit (OP)

Ok, ill buy 2 RTX Pros

Reply

[-]

MichaelXie4645@reddit

You cannot run grok 3 in just 192gb vram. No way. Also grok 3 even with reasoning isn’t all that competitive rn anyways

Reply

[-]

Anduin1357@reddit

It would be a REALLY STRONG BASELINE to have Grok 3 Mini local at the moment. With a strong offline RAG base, I could realistically go offline with it. No contest, I would absolutely spring for whatever it is that will ensure Grok 3 Mini access for keeps.

Reply

[-]

No_Efficiency_1144@reddit

It’s weaker than deepseek 3.1 though

Reply

[-]

Anduin1357@reddit

That's certainly an opinion that I will disagree with simply because I like the response style of Grok 3. Also, do you happen to have a benchmark for that?

Reply

[-]

SillyLilBear@reddit

You will need like 10

Reply

[-]

mr_zerolith@reddit

You would probably be happy with seed OSS on a single one of those, in my experience, 80-90% as smart as Deepseek R1. It needs some more nudging. That model supports 512k context and you could take advantage of that big memory of a 96gb card. On a 5090 i get 40 tokens/sec but i am limited to \~50k context with a Q4 model. You have twice the power. Another good model for a card of that size is GLM 4.5 Air.

Reply

[-]

Resident-Dust6718@reddit

For that nudging, I say up the temperature a little. When I discovered is that my laptop with a mobile RX 6650 GPU yes it’s an RX 6650 M actually out performs the RTX 4090 and some tasks with a high temperature 6 million parameter AI.

Reply

[-]

tta82@reddit

🤣

Reply

[-]

tronathan@reddit

Wow, good info. Ty.

Reply

[-]

MajinAnix@reddit

Can you investigate what speed loss you will get if you offload inactive parameters to RAM, it is possible now for MoE, if loss is not that bad then even one RTX 6000 and lots of fast ram (2xEpyc 12 channels) might be the way to go

Reply

[-]

Resident-Dust6718@reddit

For a nice fast CPU I would say something like the or nine or the RT seven series. (R9 is the RYZEN 9. RT7 is the RYZEN threadripper 7000 series. Just trying to give a little info)

Reply

[-]

AleksHop@reddit

no, every open source model can not rust

Reply

[-]

devshore@reddit (OP)

What about typescript/ javascript / nodejs ?

Reply

[-]

sciencewarrior@reddit

Open models are pretty good with popular languages like Typescript and Java. Still, that's a big investment. If you're going to run something like GLM or Kimi K2, you could first put $10 into OpenRouter and take them on a spin, see if they really do what you need before you buy the hardware.

Reply

[-]

AleksHop@reddit

dont have experience with that :(

Reply

[-]

prusswan@reddit

Pretty close if you are willing to put in RAM too. If you are spending $200 per month then it is an easy decision (now you can use more than that amount per month)

Reply

Reply to Post

222 Comments