Round Up: Current Best Local Models under 40B for Code & Tool Calling, General Chatting, Vision, and Creative Story Writing.

Posted by Iory1998@reddit | LocalLLaMA | View on Reddit | 30 comments

Each week, we get new models and fine-tunes that is really difficult of keep up with or test all of them. The main challenge I personally face is to identify which model and its versions (different fine-tunes) that is most suitable for a specific domain. Fine-tunes of existing base models are especially frustrating because there are so many and I don't know which ones I should focus on. And, as far as I know, there is no database that tracks all the models and their fine-tunes and benchmarks them against different use cases. So, I go back to you, fellow LLMers to help me put a list of the best models that are currently available, under 40B that we can run locally to assist us in tasks like Coding, writing, OCR and vision tasks, and RP and general chatting. **If you can, could you score the models on a scale from 1 to 10 so we can a concrete idea about your experience with the model. Also, try to provide the link to the model itself.** Thanks in advance.

Reply to Post

30 Comments

[-]

ArsNeph@reddit

Coding: Qwen 3 32B General chatting: Qwen 3 32B, Gemma 3 27B Creative writing: Gemma 3 27B (Note heavily censored) RP: Mag Mell 12B, Pantheon 24B, QwQ Snowdrop 32B Vision: Qwen 2.5 VL 32B OCR: Qwen 2.5 VL 32B

[-]

SkyFeistyLlama8@reddit

What, no GLM-4? I've found GLM-generated code to be better than Qwen 3 32B and it also understands user prompts better. As for creative writing, I agree with Gemma 3 27B being pretty good, but it's worth it to jump up to a larger model like Drummer Valkyrie 49B (based on Nemotron 49B). The quality increase especially in thinking mode is tremendous.

[-]

ThiagoPereiraEu@reddit

O GLM-4 se saiu muito melhor na questão user interface em python, ele faz bem legal essa parte me surpreendeu um pouco em relação aos demais

[-]

martinerous@reddit

In my experience, Gemma 3 27B is also the best for complex roleplay (scenario-based, multicharacter). Sure, you can find models with much better prose quality, but if you want something that can be both smart and controllably creative (not carried away with unasked plot twists) and not waste tokens on reasoning, then Gemma 3 is the best, beating even larger models. GLM4 also surprised me with the realistic details that felt similar to Gemma3. However, GLM4 was less stable and sometimes deviated from the scenario too much. Gemma3 is the "sweet middle ground".

[-]

AppearanceHeavy6724@reddit

GLM-4 is relatively good at fiction too. A bit dryish, but interesting. Better than Qwen.

[-]

ArsNeph@reddit

I actually haven't tried GLM personally, so I unfortunately can't comment on it. However, Instruction Following benchmarks on Qwen seem to be pretty good, so they should be hard to beat. As for code, I can't seem to find benchmarks that compare both, but it's possible they excel at different languages. I've heard a lot of good things, so it might be worth trying. OP asked for under 40B, and I only have 24 GB VRAM myself, so I actually can't run the 49B at a reasonable quant. I would love to give it a try though

[-]

xanduonc@reddit

Gemma3-27b qat is also good at vision, and it really is into roleplay to the degree of unsensoring itself lol. And i' would add qwen-30b for easy stuff but very fast.

[-]

Iory1998@reddit (OP)

Thank you for taking time to respond. I am checking te Mag Mell and the Snowdrop models.

[-]

ArsNeph@reddit

No problem :) I use Gemma 3 27B at Q4KM since the KV cache takes up a ton of memory because they haven't implemented the attention mechanism yet. It's definitely great at multilingual, and has good user preference optimization. However, the degree of censorship often has me using Mistral Small instead due to the amount of refusals. Benchmarks show it to have one of the highest rates of hallucination of any model, and I found this to be mostly true in my testing

[-]

Mushoz@reddit

Sliding window attention has been implemented in llama.cpp a few days ago. It results in a 4 to 5 times drop in KV cache size for Gemma 3. Worth installing a new version if you are using Gemma 3

[-]

ArsNeph@reddit

Thanks for the info! Unfortunately, I'm not at home, so I can't try it out yet, but I should finally be able to run it at Q5KM or Q6! That said, I use upstream versions, namely Oobabooga and Ollama, so it might take a little time before they are updated

[-]

admajic@reddit

Just trying gemm3 12b Fosowl/agenticSeek got it setup and running with lmstudio is like manusai for local use.

[-]

EducatorThin6006@reddit

Been thinkjng to do the same. So excited.

[-]

Iory1998@reddit (OP)

Is the model good?

[-]

admajic@reddit

Seemed ok. I ended up using openrouter with deepseek-r1. Came back and the whole program had crashed. The profram is still very buggy. I just found openrouter root was quicker.

[-]

productboy@reddit

Not going to follow your rating rules but will say this model has been exceptional in my testing; with a very small VPS instance; qwen3:0.6b

[-]

Iory1998@reddit (OP)

Thank you, you don't need to.

[-]

sammcj@reddit

I feel like there needs to be two weekly polls one for coding models and one for general models as this is constantly getting asked every day (not having a go at you OP, just saying it would be useful).

[-]

Federal_Order4324@reddit

Sillytavern Reddit has a weekly API/model discussion post I think locallamma could do smth similar

[-]

Impossible_Ground_15@reddit

\+1 to this u/sammcj is 100% correct.

[-]

Iory1998@reddit (OP)

I agree. This is the main reason I am asking here. We need a way to track all the models we get every week.

[-]

RickyRickC137@reddit

Instead of just tossing out 1-10 scores, which can be subjective, I say we crowdsource a ranked list of the best models under 40B for each task. Here’s my pitch: * **Share Your Faves**: Drop your go-to models in the comments with links and why they’re great for a specific task. Like, “Model X kills it at Python debugging” or “Model Y nails RP convos.” * **Rank by Task**: We compile a master list, ranking models based on what they’re best at. No generic scores, just straight-up “this beats that for coding.” * **Monthly Refresh**: Let’s keep it updated monthly in a pinned thread or a Google Doc we all edit. I’ll start the first list based on this thread’s input. Here’s the lineup based on my take: * **Reasoning**: Qwen 3 32b > QWQ 32b > Qwen 3 30b3A > Gemma 3 27b. * **STEM**: Qwen 3 32b > Qwen 3 30b3A > QWQ 32b > Gemma 3 27b. * **Math**: QWQ 32b > Qwen 3 32b > Qwen 3 30b3A > Gemma 3 27b. * **Coding**: Qwen 3 32b > QWQ 32b > Gemma 3 27b > Qwen 3 30b3A. ( I don't do coding. This is the experience of my buddies who use these models) * **Creativity**: QWQ 32b > Gemma 3 27b > Qwen 3 32b > Qwen 3 30b3A. * **Chat**: Gemma 3 27b > Qwen 3 32b > Gemma 3 12b > QWQ 32b. English is my second language, and Gemma nails convos in multiple languages.

[-]

DrAlexander@reddit

Does RAG performance fit into any of these categories?

[-]

RickyRickC137@reddit

I haven't tested RAG because I don't like the current state. I would rather wait for the model's context window to be well enough to include a whole book (more importantly my Vram to be powerful enough to handle such context length) and use CAG.

[-]

DrAlexander@reddit

Well, that makes sense. RAG is hit or miss, currently more miss though. I keep hoping to find a solution easy to set up and accurate with the available tech, but being able to run a 1M context model locally would of course make more sense. The problem is that, for personal use, you either need a substantial investment which might not be a good financial decision, or wait 5 more years.

[-]

dkeiz@reddit

man you want reddit automatic gathering data on all llm under 40B release? cant you use LLM for that purpose? Just kidding, but i guess there some private blogs there with such intention. Testing them on the other side, is extremely hard.

[-]

Iory1998@reddit (OP)

Just share your favorite models would be greatly appreciated

[-]

dkeiz@reddit

I really like quality in devcoder14b and devstral. But it all nice and good until they start circling instead of incrementing improvements in code generation. What is it, lack if context memory or bad prompting, since one-shot prompts code quality is much better then prompt with improving code? Recently i goes for one-shot coding instead of improving code prompts. But i cant say that there some model quality, its felt like problem with attention and memory managing. And thats why comparing different models extremely hard. They met same problems, and solve in same way. While code style (that could be called quality) may be completly different. And in the end of day we got lots of web-available benchmarks that tells nothing. I personally think that model allready all good. Its about the way we use them.

[-]

EstebanGee@reddit

It’s almost like we should get an llm to summarise the posts for the week. Nah, let’s just use a spreadsheet :)

[-]

FPham@reddit

Gemma-3 is surprising at the creative part. Of course it likes to reuse slop, and similes. Everything is like something to the point of nausea, like a tidal wave flushing through your brain. (did that on purpose) But its story weaving with the proper instructions (scene beats) are kind of shocking, It would go through the beats and join them very, very nicely together, creating a rather complex story, far more than any model of this size (I'm talking bout the 27b). It's not about the language (can't help uber-written slop at this point), it's the story writing capabilities. Gemma doesn't have system prompt so everything is system prompt. I tried many things, examples of style, instructions etc, and gemma works with all of it. Of course as any self respected LLM the result will be an exaggerated mimic of whatever style you sample in it, basically finding the inflection point and characteristics of the style then dial them to 11. It does work, so even just trick it with reverse -1 examples of it's own writing will work, but again, dialed to 11, almost as making fun of the style. The only way to attenuate that language would be LORA, but my attempts at that failed. I did make a Lora, but then I'm unable to apply it in WebUi, it probably due to the different architecture - I know there is a guide on google with code, but I managed to ignore it. If anyone is familiar with this part, let me know. All in all, personally I haven't found a better model of this size that can genuinely be so bendable. Yes, the result is almost unreadable for the slop, but the meat of it is actually really good. So at this moment you need gemma and a rewritting model.