Duality of r/LocalLLaMA

[-]

RedParaglider@reddit

I think people think for some wild ass reason that they can pick up a tiny model on local inference and run it like a full weight model running on half a million dollars in hardware. It would be like buying a 500 dollar E-Bike on amazon, and being irritated that you can't drag race with it very well.

[-]

Electronic-Space-736@reddit

it takes real effort to get small models to be effective, some people have worked it out others are struggling

[-]

aadoop6@reddit

Mind sharing successful recipes?

[-]

FatheredPuma81@reddit

Something I've found successful (as a none programmer lol) is telling the model to use Subagents, to test its shit after every update, and then having multiple models do full audits of whatever you're making and save all of the issues to a file. Works well enough for my use case at least.

[-]

markole@reddit

I wrangled Gemma 4 into translating to a low resource language by providing it a custom mcp server for precise translation editing and a custom translation memory tool so that it can align its outputs. Helped a lot. You can employ similar tactics for your problem.

[-]

wilo108@reddit

Would love to hear more about this set-up -- have you written about it anywhere, or could you share more here?

[-]

markole@reddit

No writeup but it's somewhat straightforward. You need to have a translation memory of sorts, my is a linux CLI that the model can call to suggest most similar translations based on the input string. It uses FTS5 in SQLite for this.

Then the model also has an MCP server to get untranslated strings and write new ones to a file. I used to have it do it through shell scripts but it was too imprecise for that so an MCP server is desired.

The third piece is just iterating on prompts and the translation memory. My memory entries have a context field beside the actual strings in which I basically give pointers to the LLM when to use what, what to never do and so on. This is an iterative process, you need to observe the outputs to see where it's struggling and then use the context to help it out.

[-]

juraj336@reddit

For me it has been all about giving it the right context.

Im using pi coding agent with qwen3.6 27b and have created extensions it can use to gather specific data. For example searxng for general search and a custom scraper for some local webshops for prices.

This way it can find its way to the data it needs and use it. Im of the opinion that this allows it to operate close to way more expensive models that have all this context inside of them already

[-]

Electronic-Space-736@reddit

it is very much about information density https://github.com/doctarock/local-ai-home-assistant/blob/main/docs/COMPRESSION_QUICK_REFERENCE.md

[-]

wren6991@reddit

Looks like you used an LLM to generate that block diagram. They're never very good at getting the right-hand edge to line up.

The approach described there looks useful but no more useful than adding a tail -n10 to the test command line in your AGENTS.md. Obviously you shouldn't be loading kilobytes of CLI tool output into context. Agents are perfectly capable of just grepping the output once they see a failure.

[-]

Electronic-Space-736@reddit

yes that is just one part of the picture, totally agree, and yes, never been inclined with the ascii art but codex maintaining documentation is a life saver honestly, for more of the picture - there is also RAG via qdrant which helps a lot, a bit of hardwired cheating, but the most effective way to get these small models to force through the request is with a reshaping loop, it does slow things down, but gets the tasks across the line

[-]

Electronic-Space-736@reddit

here you go https://github.com/doctarock/local-ai-home-assistant

[-]

aadoop6@reddit

Thanks a lot. Anything for agentic coding applications?

[-]

Electronic-Space-736@reddit

this is an agent, it is agentic and it is a chat bot, plus a bunch more, and I release plugins every day.

[-]

No-Dot-6573@reddit

What local llm would you recommend? Also the naming made me think of it being a home assistant integration at first xD

[-]

Electronic-Space-736@reddit

I do have home assistant integration in the next release.

Currently
Gemma-4
Hermes-3
Qwen-3+
DeepSeek
Various specialists

They run in dedicated lanes and the code above orchestrates as many brains as you can throw at it

[-]

TheTerrasque@reddit

I think there's also many subtle ways people can mess up the hosting itself or the coding tool setup. Like for example high temp, bad templates, wrong token setup in server, low context, even worse low context with rolling window, misconfigured context in client, too aggressive quants, badly converted model files, old binaries....

I wouldn't say it's an art to it, exactly, I've had good success by using updated llama.cpp and unsloth recommendations, but there are many ways to mess things up that won't directly fail, but will lower the quality in various ways.

Like in the other thread someone complaining that basic tool calls fail with qwen3.6 for example, which suggests something is pretty wrong with his setup.

[-]

Electronic-Space-736@reddit

yes absolutely, the more you know the easier it is, but this will become abstracted over time as things evolve, we are still pioneering here, the tools are rough, the processes are not streamlined, and the goalposts move daily

[-]

andy_potato@reddit

You know, some people have a life.

[-]

Mediaright@reddit

Then pay the money. What do we care?

[-]

Electronic-Space-736@reddit

it wasn't a personal attack Mr Potato, just an observation. I have free code if you are struggling to get a baseline set up

[-]

No-Marionberry-772@reddit

its a seripus problem and no one s making any OOTB solutions, or its not being promoted well enough

[-]

Electronic-Space-736@reddit

I think people are doing the golem with the ring lol, others flying low to make a buck, and the foundation model folk are not keen to share such info

[-]

FatheredPuma81@reddit

Weirdly enough I expected the reverse. Went from Devstral (the old like 25B model) running in Openhands to Sonnet 4.0 in Claude Code and expected it to be much better. It wasn't so I just gave up on my project and canceled my subscription before the end of the month.

Then I tried Qwen3.5 and 3.6 in Opencoder and yea wow it's better than Sonnet 4.0, actually made a working program, and its free!

[-]

Winter_Educator_2496@reddit

I've seen people say that the only thing big models have is more knowledge. The knowledge of all things obscure is a fundamental part of programming. And there is really a shaky difference between being "smart" and just having lots of knowledge.

The current benchmark results that show these small models competing with the biggest ones is very consistent with test-time leakage, as opposed to general ability. If the main model was trained somehow on the targets for these benchmarks, then it would be very good at giving this knowledge to distilled models too. Really makes you think what it says about the capabilities of these big models then.

[-]

RedParaglider@reddit

Yea, there is a reason most offshoring of development fails. It isn't because offshore developers are bad, a lot of them are actually quite amazing, but they are hobbled by the real world business knowledge of the business. They are given a HLD or whatever and told "make it happen". But that's just a somewhat filled in real world prompt that would leave them guessing.

[-]

ea_man@reddit

With Pi, using a prompt < 2k.

[-]

tiffanytrashcan@reddit

I watch people try to hand OpenClaw to a 4b model running on a phone then get confused "why is this injecting 32k tokens and breaking?"

[-]

NicolaRight@reddit

And it goes on

[-]

Intelligent_Ice_113@reddit

indirect Claude and co propaganda.

[-]

Ill_Barber8709@reddit

bUt yOu CaN't rEpLaCe cLaUdE wItH a lOcAl lLm

I've been using Qwen3.6 27b Q4_K_M in Claude Code on my 32Gb M2 Max with 64K context and this thing convinced me I could replace my Claude subscription with a full local setup (especially since they're cutting off Code from the Pro plan).

Qwen3.6 is crazy good. And it's not even a coder model. Can't wait for their future models.

[-]

boutell@reddit

Hello, fellow 32GB M2 Mac 27b user! You have the Max, so your memory bandwidth is twice mine, which is part of why it's working well for you. But I would be interested to see your stats and your llama server command line, if you can spare the time. Thanks!

[-]

Ill_Barber8709@reddit

Alright. So for comfort I decided to use the M2 Max as server only, and do everything else on a base M4 Mac mini.

I push the GPU memory limit to 28GB to get some room for context.

I use LMStudio to serve the model, because it really makes everything easier to configure. KV Cache is enabled. 65536 tokens of context. Preserve thinking is enabled. Everything else is default.

Unfortunately I can't really give some stats (TPS) but I have to admit it is very slow. You can't really vibe code like you would do with Claude. So you have to do something else while the model is working.

I use the Unsolth fix "CLAUDE_CODE_ATTRIBUTION_HEADER" : "0" in the Claude Code settings. Without this, Claude can't use the KV cache and things get even slower.

I didn't try MLX version of the model yet.

I hope it helps.

[-]

leocus4@reddit

How many tok/s do you get? Is it fast enough for everyday's work?

[-]

slvrsmth@reddit

No. `Qwen3.6 27b Q4_K_M` gets me \~22t/s on M5 Max / 64GB system. I'm sure my configuration is far from optimal, and I could push maybe 30t/s on that model. That is still unworkable.

`35B-A3B` works at >100t/s. That's good, that's something you can work with. But the output is... it leaves much to be desired. The code often works, but the result just looks sad. Example from earlier today - qwen decided to repeat all the allowed letters in a regex in both upper and lower case variations to achieve case insensitivity. ALONG WITH the case insensitive flag. Belt and suspenders. Yeah, after coaxing and prodding it re-wrote the monster regex down to something reasonable. But if I have to prod the robot along every step of the way, I might as well write the code myself.

Besides, the device running those models cost more than three years of claude max. That's about the depreciation term of the device from accounting perspective. All to achieve sub-par results.

[-]

boutell@reddit

I'm here for this perspective too. And yes, 35b technically works but if you actually read that thinking trace you will want to slap it over and over LOL

[-]

Quirky_Inflation@reddit

35B-A3B is garbage for coding, it doesn't follow plans. Gemma 4 31B is good, still have to try Qwen 3.6 27B but I have some expectations into dense models for coding.

[-]

michaelsoft__binbows@reddit

"Theyre cutting off code from the pro plan" i'm sorry, WHAT.

[-]

droptableadventures@reddit

https://simonwillison.net/2026/Apr/22/claude-code-confusion/

Simon Willison's written a blog post on the whole story with this (yeah, there's been so many twists in that story it's a full blog post).

[-]

michaelsoft__binbows@reddit

cool, thanks.

[-]

redmctrashface@reddit

No they are not afaik. They removed it for few hours and put it back.

[-]

Evanisnotmyname@reddit

probably testing the reaction.

When Anthropic stood up for safety I really backed them. As time has gone on all AI co's just seem shady on the billing/plans/APIs/astroturfing front..it's still tough to keep backing them.

[-]

xamboozi@reddit

Yea, I'd rather just sleep well at night knowing my data isnt leaking and I'm not supporting the unethical stuff our government is doing

[-]

redmctrashface@reddit

I agree with you, especially when you see the hype on mythos vs. the official paper released. Opus 4.7 is shittier than what Opus 4.6 used to be. It doesn't smell good for end-users imho.

[-]

thrownawaymane@reddit

They changed all their pricing info on their site to remove code from the Pro plan as well.

[-]

stoppableDissolution@reddit

Not sure, tbh. They seem to be well out of compute and not really capable of serving more users anyway, otherwise there would have been no price gouging

[-]

kaeptnphlop@reddit

They are only surviving because of massive investments. Not because of their own revenue streams.

Price gouging was inevitable

[-]

CalBearFan@reddit

It's not price gouging if they're charging realistic rates now. Large increase yes but gouging is charging massive amounts over cost during a external shock like a natural disaster

[-]

WinDrossel007@reddit

Now I need to check mode local models 😄

[-]

Intelligent_Ice_113@reddit

I believe the bots are starting to try to convince us to eventually pay for that $100 subscription.

[-]

NNN_Throwaway2@reddit

Crazy the amount of cognitive dissonance in this thread/sub.

"small models are totally useful bro you just need to jump through a bunch of hoops to prompt them right oh and its a skill issue of you believe they are actually almost on part with SOTA, which they are by the way if you prompt right."

Meanwhile, clearly no one has actually read the first thread, which is being presented here out of context for the purposes of a braindead meme that everyone can seal-clap to.

[-]

smirnfil@reddit

Don't forget "Cloud costs a lot. you could save heaps of money by bying 5090 and running models locally" vibes. The funny thing that both statements are true - local models are at the level that you could use them for real work, but there are noticeable gaps between them and the current SOTA.

[-]

EvilGuy@reddit

For me the equation is like this. I can run 3.6 27b on my 3090 and its actually decent but Deepseek 4 flash exists and is better than what I can run and they are basically giving it away... and my power isn't free.

So yeah until the equation changes I am probably going to be using non local LLMs for the near future, even though I find them cool / interesting and I like owning my data etc.

Anyone else in the same boat?

[-]

Pineapple_King@reddit

I just developed a warehouse management software with graphical interface, QR code printing and order management in the past 24 hours, from idea to bug polished.

qwen3.6 for the win.

[-]

laffer1@reddit

With all the throttling, we have no choice but to run some workloads locally.

[-]

Memexp-over9000@reddit

I have been using qwen 3.6 27b, and yes of course it has limitations. It's a freaking 27b model. It's a skill issue to expect it to be competitive with 1 trillion+ params. But it at the same time can achieve outputs almost at par with the trillion+ models IF AND ONLY IF you know how to harness it's power by architecting your workflows efficiently. Basically you cannot hand over the creative aspects of your projects to the AI, just the grunt work part.

[-]

LeonidasTMT@reddit

Part of it is the posts that keep hyping it, qwen 3.6 surpasses model xyz in coding benchmarks etc

[-]

slavetothesound@reddit

And the people say in g they are getting work done never share details of what that work was, how complicated it was, or what their harness looks like and how long it took them to set it up.

They just set the base expectation that a model does work for them.

[-]

socialjusticeinme@reddit

Anyone doing real software engineering work with this stuff knows where even opus will fall flat on its face and shit the bed. Anyone claiming they’re getting work done with local AI is probably making yet another flappy bird clone or some scam AI website to show off their cool new X that you just have to pay them for.

Now with having said that, I do use local AI even for real work, but it’s stuff like bash script writing, simple python/javascript, and to remember Linux commands - not the actual heavy lifting. That itself is so useful that I have a dedicated local AI box.

[-]

RedParaglider@reddit

I have a 58 year old engineer friend that loves local models, so IDK what you are talking about. He calls it code completion on crack. I set him up with codex, and he's like meh. He still just pastes his code file into webui, tells it explicitly what to do, then copy pastes it back in old school.

I thought he was a madlad for not using a TUI, but turns out he knows what every fucking thing does in his LLM written code lol.

[-]

Audratia@reddit

To be fair, I do the exact same thing and I'm 32 with 10 year of professional developer experience. I do it using both local models at home and our enterprise grade claude subscription.

Honestly, its just the text ui aspect for me. Really does not work with my flow. Now web ui tools for agentic dev. Spot on.

[-]

Fantastic-Balance454@reddit

Not only will he know more about his code, but also complete context of the code compressed into single output will outperform agentic workflow majority of the time in terms of code quality and precision.

[-]

LeonidasTMT@reddit

Yep Qwen3.6 for me is not there yet for complete agentic workflow. I still need to baby and guide it step by step. Otherwise it tends to go off tangent

[-]

TheTerrasque@reddit

I am in the process of testing out Qwen3.6 as a coding engine, after having a lot of success using it as a personal assistant for a while. So far it's impressed me. I'm using claude code and codex at work, and while they do crap out now and then they're pretty good and getting better. But they do run out of tokens quickly on the plans we have atm.

Anyway, even those big models fail pretty regularly, and this is a small model, so I think my expectations are tempered a bit. Anyway, it's been nice for smaller things where using claude or codex feels like a waste, and to push it a bit I had it build a small mcp server - I wrote a bit about it here.

I think I can replace maybe 50-90% of what I use claude / codex for today with a local model, which will help a lot, and for smaller things like that mcp it can do the work entirely on it's own.

[-]

Ill_Barber8709@reddit

It's not that hard to work with Qwen 27b.

My current workflow: - Use Claude Code - Ask to create a plan with the most detailed steps possible - Reset the model and ask to work on step N - Verify everything is working and update the plan. - Repeat with step N+1

So far I've been working on rather complex Vue projects with good results (but NGL very slowly) with the unsloth KV cache fix (one line in the settings file).

[-]

HiddenoO@reddit

It's a skill issue to expect it to be competitive with 1 trillion+ params.

Well, clearly a lot of people on this subreddit have skill issues when you regularly have upvoted comments suggesting that you can do the same with those small models you can do with larger ones.

But it at the same time can achieve outputs almost at par with the trillion+ models IF AND ONLY IF you know how to harness it's power by architecting your workflows efficiently. Basically you cannot hand over the creative aspects of your projects to the AI, just the grunt work part.

You can also achieve outputs on par with the trillion+ models by writing everything by hand, so that's not really an argument for anything.

Way too many comments in thus subreddit act like you just have to "prompt better" to get similar results, when in reality, you have to put in significantly more time and effort to get comparable results.

[-]

Menotyouu@reddit

yeah i find it that you really need to be precise with what you want for it to do, and even then may need to review because it sometimes makes assumptions it shouldn’t. But i think that’s great, it forces you to be more mindful and think a bit deeper compared to just asking Opus to implement C

[-]

relmny@reddit

Glad to read someone is able to understand that a, probably, > 1tb model with many tools, is way bigger than a 27b/31b model. And what are the implications of it.

AI is really making is dumber if we aren't able to understand something as basic as that.

[-]

LeucisticBear@reddit

Did you read the local model complaint OP? He was handing pretty general tasks to Gemma and hoping it would just work. Even a very basic "plan with gpt or Claude, execute with local" would work better. Local isn't smart enough to do all the thinking, particularly when you need it to do advanced reasoning and make assumptions. It's absolutely good enough to treat as a subagent if you give it a well defined plan. That's the big discrepancy.

[-]

viperx7@reddit

Well I guess VRAM matters local models when you have 12GB VRAM vs when you have 96 GB VRAM are 2 different things

[-]

ea_man@reddit

I think that having a 15K contex prompt just to start in the harness + LLM that have been trained to use the tools the harness uses may be part of the magic.

Vertical integration.

Either you accept that small local LM are for smaller tasks or you at least need to use a very structured prompt to tame a mid size project.

Here there's people (not judging, I've ben there too) that cry because Continue doesn't use the XML tools of QWEN, Pi with A3B isn't as autonomous as Cloudcode + Opus.

[-]

Kodix@reddit

Harnesses are definitely a *huge* part of the difference. I've found normal Qwen3.6-35B to be a bit of a dummy. Since I started using it in Hermes, I've had nothing but good results. It's no Claude, but it is genuinely a useful assistant.

[-]

ProfessionalSpend589@reddit

I personally think anyone who downloads the files for a 1T model to load them on a low end 12GB VRAM GPU requires a gentle explanation about how to use computers in general.

[-]

Lesser-than@reddit

math is hard sometimes /s

[-]

MasterLJ@reddit

Qwen3.6 27B is incredibly capable.

GLM-5.1 is too, it's a bit more expensive to run.

It's all so much cheaper than Anthropic though as I can pay by the hour for GPU compute that shuts down when not in use.

[-]

ICatchx22I@reddit

As always, the answer is: it depends.

[-]

BlobbyMcBlobber@reddit

Local models can work, but it's not easy and instantaneous to set up, so this about sums up the entire story. People expect to just load a model and get a local Opus 4.7 with zero understanding on harnesses, optimization, task alignment. So they get frustrated and post about it.

If you stick it through you can can get great results but this is a skill with a learning curve and not a product from OpenAI or anthropic.

[-]

Late-Assignment8482@reddit

I feel like part of the cause is differing dev needs. I'm a sysadmin for whom dev work is second-string--Python, Bash, sometimes Swift--so most quality local LLMs can do it and can ace it if I build in guardrails.

If I wanted to make a Mac/open source cross platform app for some casual use, I'm sure they could do that too.

If I'm trying to build the main product app for a startup, higher complexity and stakes.

[-]

Karnemelk@reddit

If you have x hours to implement feature y. Would you use claude or a local model if money wasn't an issue?

[-]

HornyGooner4402@reddit (OP)

If money wasn't an issue, I would've bought 10x H100 to run GLM or Kimi locally

[-]

No-Selection2972@reddit

Even m2.7 is good enough for me

[-]

krzyk@reddit

Yeah, after reading one post, I'm convinced that I could upgrade my 3060ti into 3090 and install some local model.

Few hours or a day later I read that it is wrong and I can expect it to do agentic work similar to a cat walking on a keyboard. And I ditch the idea of 3090.

I think I need to find some cloud gpu comparable to 3090 and setup llama.cpp there to test it out.

My laptop has only 6GB RAM, 3060ti has just 8GB, so I can't compare it on them (I think).

And I hate iOS, so any mac minis are out of the question (even if I could find one) (well, I could borrow air from my wife, but I don't assume I'll get good comparison working on a 16GB unified RAM :)

[-]

xamboozi@reddit

Idk if you know this but cloud AI providers have a very large budget that can be allocated to make sure public discourse remains profitable for them.

[-]

entsnack@reddit

everything is a conspiracy when you don't know anything

[-]

WithoutReason1729@reddit

Your post is getting popular and we just featured it on our Discord! Come check it out!

You've also been given a special flair for your contribution. We appreciate your post!

I am a bot and this action was performed automatically.

[-]

noctrex@reddit

+1 For the quantization that people forget to mention they use.
Was trying out Qwen3.6-35B-A3B with a Q4 quant, so that i can have it fit in my 24GB VRAM, and it was looping, was the repeating itself, and was failing at tool calling half the time.
Thought the model was trash, but then downloaded the Q8, and offload it , and it's working perfectly with everything.
Of course, it's slower than having everything in VRAM, but damn it gets the job done

[-]

TheTerrasque@reddit

which q4 quant were you using? I'm using unsloth's q4 xl quant and have only seen looping once, and never had a problem with tool calls.

[-]

Bulky-Priority6824@reddit

You just gotta know how to use that thang!

[-]

drwebb@reddit

I'm with the skill issue people, and quantity of slop is a quality of it's own.

[-]

Cool-Chemical-5629@reddit

And look which post has more upvotes. 😂

[-]

Scared-Tip7914@reddit

This is lowkey because of the difference in quants that people are running, I mean almost no one mentions if their qwen3.6-27B is a Q2 or Q8 in their posts about “omg local models are replacing my Opus-4.whateverthef*ck workflow”. I personally am running qwen3.5-35B, Q4 from unsloth with cline and find it to be amazingly competent, BUT its an extension to the big proprietary models, when you dont want to burn tokens, NOT a full workflow orchestrator. I will say what I always do, plan with the likes of GPT/Opus/Sonnet, execute with a local model.

[-]

solestri@reddit

I've noticed that this is a huge problem with pretty much any use case of LLMs. "This model sucks/is amazing/excels at this thing/can't do this other thing"... and zero mention of any of the other variables that are almost certainly contributing to that experience.

[-]

tednoob@reddit

You don't think the Qwen3.6 moe is an improvement on the 3.5 version?

[-]

Scared-Tip7914@reddit

No it definitely is, I was maybe a bit unclear with the phrasing, I will be upgrading to it probably at the end of this week, I just like to do some thorough testing before I switch my main workhorse over :D, but so far it surpasses 3.5 in the cline/opencode style agentic coding tasks

[-]

a_beautiful_rhind@reddit

"I used local models for my code thing" vs "We ran terminal bench"

Ok.

[-]

False_Process_4569@reddit

Reality, which is subjectively experienced by each of us, gets more distorted and we'll all agree less on what is real as we get closer to going through the technological singularity.

[-]

FoxiPanda@reddit

Honestly I feel both. Some days local feels like magic, and some days local feels like I'm talking to a lobotomized half-sentient brick.

There's also a shit ton of variables at play:

What anyone is trying to accomplish varies wildly day to day.
What harness people are using matters a LOT and half the time we don't even factor it into a response or post.
Model quantization varies wildly. One person's experience of Qwen-3.6-27B might be completely ruined by a IQ2 quant and another person running a Q8 has a phenomenal experience.
Prompts actually matter. Like for real. "Can you do better" or "fix my docker" are not great prompts people.
Half the people writing code probably don't even create a PRD/Architecture document/even a code plan at all before they just yolo into it.
People use hilariously over-optimistic speculative decoding or presence/repetition penalties and are like "omagerd my model gets into loops. sucks so much!"

So yeah. People are going to have wildly different experiences. It's the nature of the beast. It makes the signal to noise ratio not great.

[-]

HenkPoley@reddit

lobotomised half sentient brick

Hey, that’s the name for my next model..

[-]

LeonidasTMT@reddit

Sometimes it's just the luck of the seed. One day I can get qwen 3.5 to run everything without having to intervene, another day it keeps failing the tool call or getting stuck in loops.

Same task same settings, just luck.

[-]

LegacyRemaster@reddit

then you make a post where you can run deepseek v4 flash via gguf saying that the changes (although ugly) to the code (still working) were made only with LLM and they delete it..

[-]

Ok-Measurement-1575@reddit

Posts with considerably higher effort have been deleted in the past.

[-]

dsartori@reddit

These conversations are difficult because the range of possible scenarios is too wide and people’s expectations are too different.

I exclusively use local AI for my personal and professional coding work, with good success but I also use these tools in a certain way, on a certain set of problems.

its a big field and expectations are all over the map. I don’t doubt either poster’s experience.

[-]

cagriuluc@reddit

The bigger the model, the less you have to worry about.

If you are using a cheap-to-run small LLM, for it to compete with expensive, big and capable models, you must be the “intelligence” that is lacking in the small one. You need better engineering, better prompting, better understanding of how stuff works.

[-]

havnar-@reddit

I just had qwen MOE quickly do a get merge of 1 file from another branch to my local one.

It started to do a full merge right after it did what it had to. So stopped it. Then it was so confused it went in circles for minutes. I watched it do its psychosis and pulled the plug when it stopped being funny.

[-]

jacek2023@reddit

the real duality:

- people interested in local LLMs, running them, testing them, finetuning them

- people who hate local models, are here for cloud models, they are interested in "DeepSeek/Kimi/GLM cloud is cheaper than Claude Code" and in benchmarks/leaderboard

The second group is on the rise

[-]

m31317015@reddit

Please for god sake just stop hyping models into the local superclusters area, those who wants one click done out of local models should stop. It's literally people giving the vaguest instructions expecting extremely detailed crafts and rage about it when the first revision doesn't work immediately as they thought.

[-]

eat_my_ass_n_balls@reddit

People’s tolerance for models fucking up is extremely high

[-]

Zarbokk@reddit

27B needs good instructions, and what to think about, and what not to miss.

Bigger model think about what you need and not to miss by themself.(mostly at least)

I tried some auto research things with 27B. That failed miserably.

[-]

chikengunya@reddit

The thing is: Yes, Qwen3.6-27B is damn good for use in a coding cli (both opencode and pi.dev work really well), but: you have to think like a programmer and give it clear instructions. Of course opus 4.7 understands 'less precise' prompts better. Example: I had a PDF with questions and answers and wanted to turn it into an interactive HTML Q&A. If you just give the 27B model the PDF and say 'make me a Q&A HTML from this', it will struggle because the real question is: Can you easily extract the Q&A from the PDF's container format, or should you do it via OCR instead? In my case, the latter turned out to be the more robust solution. If you give it clear instructions, you get a very good result. Opus can of course handle more complex stuff, but how you prompt and what strategy you use is extremely important. I can totally understand why many people say the 27B is a solid opus replacement, it is for me too, but obviously not for ultrahard coding tasks. For normal day-to-day problems, though, the 27B is damn good. And since it came out, I've been using my 4x 3090 system a lot more, which shows just how usable it really is.

[-]

nikhilprasanth@reddit

Both cases can be true at the same time. It's not fair to expect a model with 2.7% the size of a 1T model to behave like the Trillion sized model. The smaller models are getting way better at tool calls. Use the bigger models to create structured plans, break them down to manageable chunks. Feed these to smaller ones, they will make mistakes for sure, debug them with bigger ones again, pass the feedback to the smaller one. Rinse , repeat.

[-]

horeaper@reddit

I was just reading those posts haha🤣.

I think they both have a point, for me currently I'm sticking with comment prompt autocomplete (FIM). It helps alot on boilerplate code and common algorithms (so I don't need to google everytime). Also it force me to write clear comments, which is a good habit anyhow.

[-]

Lissanro@reddit

I think current local models are quite good. I mostly run Kimi K2.6 and also GLM 5.1 if the former gets stuck on something or for cases when I know GLM 5.1 is likely to be better. But harness is important as well. It needs to support native tool calls for the best results and also important to know how to use it, it only comes with experience. I am mostly using Roo Code and for some specific tasks, custom built agent framework.

I sometimes as well use small models for simpler tasks, for example I found Qwen 3.6 27B very fast and also capable of processing video input, making it the best for use cases that need this. If I still need larger model capabilities, I can make it describe the video in the format I need and then let the other model continue the task. Also, small models are quite good for quick iteration that involves edits of small to medium complexity, and at batch processing files, such as translating many language files in json format.

Overall, I do not feel that I miss anything by not using the closed cloud models. Also, having local setup allows me to work on projects that restrict me from sending data/code to a third-party, and of course to have full privacy for my own tasks as well, so I do not have to worry about leaking any personal information if I keep everything local.

[-]

FriendlyUser_@reddit

Working with 3.6 27B, Omlx on a 48 gb Ram MBP m4 pro with 3bit turboquant and as DWQ model with 256k context. 400-500 tokens inread, 100~ output.

I was able to update a 200 class java project abd casually asking backend information from another nodejs project without issues, first shot.

What is most important? Git structure, agent files, skills.

[-]

ghulamalchik@reddit

Depends on your hardware, and the language you code in. Some languages are more trained for than others.

[-]

Durian881@reddit

Both can be true at the same time. When using any tool, it's good to be mindful of its limitations and compensate/adapt accordingly if we want it to be effective.

When using a local model, providing more specific prompts and context (e.g. documentation) will help greatly. But we also need to be mindful of the limited context window, and not overload the local models with unnecessary information.

A stronger cloud model with huge context can definitely achieve more via brute force and much more training data.

[-]

Due_Duck_8472@reddit

It's just obsession in an echo chamber, and sunken cost fallacy.

If you spent all your life savings on a LLM rig you will never admit it doesn't work.

Locallama was created by people to run sillytavern, to run uncensored models, to roleplay subjects/kinks/perversions deemed illegal or highly sensitive to pass on to API services.

[-]