Comparison of Qwen 3.6 and Gemma4 (MoE and Dense models, Q4_K_M), generating a moderately complex MySQL query, only one produced acceptable results

Posted by bgravato@reddit | LocalLLaMA | View on Reddit | 43 comments

I tried Qwen3.6 35B A3B MoE, Qwen3.6 27B Dense, Gemma4 26B A4B MoE, Gemma4 31B Dense. In all cases I was using Q4\_K\_M and thinking mode enabled. I also tried Qwen3.6 27B Dense in Q6\_K version. Same prompt for all. The only one that produced a working query, that did exactly what was asked, was Gemma4 31B Dense. Not even Qwen3.6 27B Q6\_K was able to do it. Gemma4 also did it considerably faster than Qwen3.6 27B. Given all the comments I've seen in the past weeks I had high hopes on Qwen3.6, but for now it was a bit disappointing... What has been your experience with these models for generating MySQL queries? Next I'll try some PHP code generation... I hope Qwen3.6 does better there.

Reply to Post

Reply

43 Comments

[-]

Significant-Yam85@reddit

What settings do you have for Qwen 3.6 27B. I would set all parameters according to the recommended except repeat-penalty which I set to 1, this makes the thinking clear and efficient in my tests. Please post full llama.cpp launch commands and hardware so we can follow along and help answer your questions :)

Reply

[-]

ai-christianson@reddit

You are asking for the launch commands, which is a good start. To make this reproducible, we also need the actual prompt and the DB schema structure. Quantization (Q4_K_M vs Q6_K vs FP8) and temperature settings heavily influence code generation reliability, so a fair comparison requires identical prompts and full parameter disclosure for every model tested.

Reply

[-]

bgravato@reddit (OP)

I used the exact same query in all cases. Parameters used were the ones recommended for the model. This was on an actual production database, so I rather not disclose it publicly, as it may contain sensitive information, sorry.

Reply

[-]

Makers7886@reddit

When I see the retort of "parameters used were the ones recommended" to this question to me it makes it more suspect. Qwen3.6 has 4 modes with recommended parameters along with thinking on/off and preserve thinking on/off options. So I would expect someone to show their parameters because it shows which of the many possible variations they used. Not saying it makes someone like myself simply assume you didn't use proper parameters and just think you did (more common than not). For both qwen3.5 and 3.6 I would bench all four modes against a problem/project end point and determine the best one. Results varied quite a bit based on the mode. For my purposes "General Instruct" mode is my goto with thinking off and preserve thinking on.

Reply

[-]

bgravato@reddit (OP)

Sorry, that was not my intention... Here's some more details... I had thinking on in all cases. For Qwen models, I used Temp = 0.6, Top K = 20, Top P = 0.95 (the rest off/neutral), which is the recommended settings for "thinking coding", as stated here: [https://huggingface.co/Qwen/Qwen3.6-35B-A3B](https://huggingface.co/Qwen/Qwen3.6-35B-A3B) and here: [https://huggingface.co/Qwen/Qwen3.6-27B](https://huggingface.co/Qwen/Qwen3.6-27B) (they recommend same parameters for both models). Preserve thinking was on, though for a single query that shouldn't make a difference, right? For Gemma the recommended parameters are Temp = 1, Top K = 64, Top P = 0.95 for all cases as per [https://huggingface.co/google/gemma-4-31B](https://huggingface.co/google/gemma-4-31B), although I just checked and I think I used different settings... (by mistake, not on purpose). I while ago I was experimenting with using Min P instead of Top K/P and I think I loaded those settings by mistake. So I might have actually used Temp = 0.8 and Min P = 0.05 I re-run the same prompt again on Gemma4 31B. I ran it 3 times, each with different settings: 1. Temp = 1, Top K = 64, Top P = 0.95 (recommended for Gemma) 2. Temp = 0.6, Top K = 20, Top P = 0.95 (recommended for Qwen thinking coding) 3. Temp = 0.8, Min P = 0.05 (my personal preset from a while ago, when I was experimenting with using Min P instead of Top K/P) Parameters 1. and 2. produced unsatisfactory SQL query, similar to the ones produced by Qwen models, that failed in the same way. Parameters 3. produced satisfactory result, matching what I got on my previous run, which suggests that was in fact what I used before. I retried Qwen 27B, but now with those settings using Min P (Temp = 0.8, Min P = 0.05). It didn't make a difference though, it gave me the same results as with the recommended settings. When I have the time, I'll try re-running more tests, with different settings and also give it a few repetitions at each setting for consistency/repeatability.

Reply

[-]

Makers7886@reddit

No worries thanks for the follow up. It's been a bit since I did my own capabilities benching on this model but I also tested it via a harness - which is where I found instruct general mode to work best. Preserve thinking probably wouldn't make much difference in your specific tests. It may be worth giving the non-thinking mode + instruct settings a try (mind you I see some here say they need thinking on but my personal testing shows the opposite). Also I've only used both models bf16 and unquantized cache and didn't test quants. I do something similar when I have a specific use-case/problem/project with testing varying models against it. No model has been the magic bullet and your use-case may indeed be best with Gemma4 31b. I actually probably should revisit some tests since I only gave it a look day 0-ish.

Reply

[-]

bgravato@reddit (OP)

Thank you for your comment. I don't have a dGPU. I'm using an AMD APU (Ryzen 8600G with 32GB of shared DDR5 RAM), so running tests, especially on dense models, can be painfully slow. I'm talking about 10-40 min of thinking sometimes and 2-3 tokens/sec. MoE models are a lot faster. I'm also curious to try out MTP models I was just surprised by the results on this specific use case. I was expecting Qwen to do better, especially when running Q6 vs Q4. I ran Gemma more out of curiosity to see what would be the outcome and I got a surprise. It might have been just a fortunate coincidence that it did better on this specific case, but I seem to have hit some soft nerves on some anti-google paladins here... I wasn't expecting that... I want to get proper GPU for the future, but at current hardware prices and availability I don't think that's going to happen any time soon unfortunately...

Reply

[-]

Makers7886@reddit

Oof yeah that's going to make testing painful but also you'd benefit from gemma4 being more succinct in general. I would for sure try instruct mode/no thinking with the qwen model before you toss is or make a final decision as it'll bring it closer to gemma4's token usage per answer. Also not all quants seem to be equal. So many random reports of people saying q5 beats q8 or some variation of that. It's partially why I run in bf16 and unquantized cache to eliminate "outside" factors. It may be worth downloading a different version/quant and also make sure your inference engine is setup properly. For example vLLM needs certain flags otherwise qwen3.6 flops.

Reply

[-]

bgravato@reddit (OP)

Oh I'm not tossing Qwen. My next tests will be boilerplate/CRUD PHP code generation, for which probably no-thinking/instruct and MoE models might be just fine. I also need to do some refactoring (bootstrap 4 -> 5), for which I'm curious to see how these models will handle it, but that's not a priority right now, so it may take a while to get that tested.

Reply

[-]

BoogerheadCult@reddit

It is the llama.cpp launch parameters, it has nothing to do with your script. I start to think either you are a Google shill or just an idiot who doesn't know what it is talking about.

Reply

[-]

bgravato@reddit (OP)

thank you! your comment was really helpful and respectful.

Reply

[-]

PhilWheat@reddit

What's your harness? The more I work with these, the more the harness seems to matter.

Reply

[-]

bgravato@reddit (OP)

No harness. I was just using LM-Studio and its built-in chat. I've been learning how to use VSCodium + continue, but I'm still very new to both. I've been coding and working with MySQL/MariaDB for many years, but still quite a bit old school... using only Vim (plus some vim plugins). Now I want to start using a Local LLM to help speed up some tasks. I thought about staying with Vim or switching to neovim and go with something like CodeCompanion or aider, but VSCodium + continue seemed like an interesting option too. I've added a vim-like add-on to vscodium which makes the transition easier, at least on the "editor" part. Anyway, for the task at hand for my "test", I just needed a query to generate some stats (with a couple of nested selects, a union, a few joins and some grouping), based on the relevant tables from a DB. For this, no other code context was needed, just the db structure and generate a fairly complex query from there.

Reply

[-]

BoogerheadCult@reddit

Qwen has different set of defaults for different tasks, sorry but I think you are just an idiot who doesn't know what it is doing, hence rendering your little "experiment" useless.

Reply

[-]

bgravato@reddit (OP)

Thank you. I think you are a condescending arrogant snob. Who just likes to pretend you're superior to everyone else and knows-it-all kind of idiot.

Reply

[-]

PhilWheat@reddit

You'll have tons of suggestions. I've tried quite a few, but I'll suggest taking a look at OpenCode. It's pretty easy to get set up and try but it isn't by any stretch the only one with that claim to fame.

Reply

[-]

bgravato@reddit (OP)

I still don't trust AI to mess around with my files or run any commands on my shell. I prefer a more human-in-the-loop kind of approach where I review and have to manually approve any changes... VSC+continue seemed to fit he bill here...

Reply

[-]

Qwoctopussy@reddit

there are lots of solutions for sandboxing a coding agent i used docker for a bit but it felt too heavy; these days i use lima to run a linux vm and bubblewrap to restrict permissions to a single directory i made a bash script called pi-sandbox to run pi (the coding agent i use) in bubblewrap

Reply

[-]

bgravato@reddit (OP)

sandboxing doesn't eliminate the fact that it can still screw up the files that belong to the project it has access to... Even if sandboxed, I still prefer to manually review and approve any changes before they are committed.

Reply

[-]

Qwoctopussy@reddit

true, i use git and roll back whenever i dont like the result requiring manual review of each edit isn’t a bad approach, and it would be easy for me to make a custom extension to do that (pi is kinda like the vim of coding agents — minimal and highly customizable by just telling your clanker to add an extension) but i haven’t done that at the moment

Reply

[-]

DonkeyBonked@reddit

My best results for xode that I can accept has has been with Qwen3.6 27B UD-Q8_K_XL using fp16 k and turbo-4 v But, I will add that Gemma-4-31B is reasonably close and has better inference. Plus it works better with TurboQuant and MTP. I have to babysit it more, but the only one that balances precision (not nearly as good) with tolerable speed for me on a large code base, over 512k context, has been Qwen 3.6 35B, but I'm still working on trying work on this. I don't consider any of this final for me and I still haven't tested modifying Gemma 4 models. I'm running both on my own test build of llama.cpp which is a mix of main and Tom's build.

Reply

[-]

Inevitable-Name-1701@reddit

I'm using the 27B fp8 version at the moment, I'm sure this model will eat all my web development work for breakfast.

Reply

[-]

bgravato@reddit (OP)

I don't have enough RAM to run fp8 unfortunately.

Reply

[-]

kivaougu@reddit

Why would you compare the same quant and not try to match the actual size of the quant? 1,5gb of difference there

Reply

[-]

bgravato@reddit (OP)

what do you mean?

Reply

[-]

kivaougu@reddit

Nevermind. I totally glossed over "I also tried Qwen3.6 27B Dense in Q6\_K version." I just mean that if you need to quantize wouldn't you want to give both models the same amount of vram for weights. Dense gemma 4 has more params. To be the same size as the dense qwen it would need to be at a slightly lower quant e.g. gemma Q4\_K\_M vs qwen Q4\_K\_XL

Reply

[-]

datbackup@reddit

But i mean some quants even at the same bpw can give much different results… And furthermore the same quant can give different results depending on inference settings… And likewise the harness can make a big difference … So when you tell me Qwen 3.6 27B Q6_K I’m obviously gonna ask you “which one exactly (out of the hundred+ on hf) and what inference settings and what harness?

Reply

[-]

Comfortable_Ebb7015@reddit

I have a similar use case. But my process is agentic. I give the model a table list and the model can ask for table description, joins, columns, before creating the query. I am running it on my enterprise laptop, so I can only use MoEs. Both qwen 3.6 35b and Gemma 4 26b succeed in the task, but Gemma has more success rate. Gemma is slower in t/g but is way faster in returning the result, so I prefer it. Tool call also is always spot on for both models.

Reply

[-]

fasti-au@reddit

Words and y the tic don’t work. On sense you get language bit in moe you get patterns so moe is all dot points paragraphs. You can’t give code moe really well. Qwen gets 4 or 8 moe rounds but if all your doing is adding noise the. It’s not good. Quote schema describe in code blocks and a mermaid of workflow dot point and the wiki md style ths with caps a Mandatory and the. State machine with a review step and rubric or compile lint etc then kill context and repet see if it picks up a flaw. Don’t fix just clear change the mixing oue Price and iterate. Is better. Dense is 1 hit all the things best match. Moe is fill hope run 3 b. Check result and anothe thier sep out weak for stron run revues Thst review is moe effectively having its own pattern match where open ai hndball from about 7 steps by my hue as there’s a first prompt rewrite and decide think rank then they seem to have a memory full for the biz words I. Your first Paragraph and the. You get routed to mow manifold and loop that a couple of times and it seems to have. Rename reflect reveal moment. Codex is the opposite. You on have one place but it doesn’t really talk it’s a more hard dinstruct Diesnt really matter once they realise they have the wrong animal class they will fall apart. Right now I know there’s ways to solve their issues and I have proof but they don’t want wins they want transactions and businesses to entrench and they already own the businesses

Reply

[-]

lempod_unofficial@reddit

This is how I typed when I first switched from Qwerty keyboard layout to Colemak. Only I'd spend the time to check (most) of my errors. Have you switched keyboard layouts recently? I'd appreciate you fixing your message because it sounds like something valuable to learn Thanks.

Reply

[-]

Zestyclose_Leek_3056@reddit

What the literal fuck did you even just say

Reply

[-]

bmengr@reddit

Do you have bearable token speed when using the dense models? What's your hardware?

Reply

[-]

bgravato@reddit (OP)

No, painfully slow, but I let it run in the background while I do other things...

Reply

[-]

duhd1993@reddit

Gemma may have more knowledge in certain areas. But agentic capability is the most critical metric for a coding assistant. You do not use it just by asking a question in the old ChatGPT way. In terms of true agentic capability, Qwen is the clear winner.

Reply

[-]

bgravato@reddit (OP)

>If you just want to ask a question about an SQL query, why not just ask free GPT/Gemini models? Mainly for privacy. But also free models can have some limits of usage. Yes I did use a chat on LM-Studio, but for this specific use case: generating an mysql query for which only the involved db tables structure was relevant, I don't see why a fancier setup should matter... For assisting in PHP code generation/editing, I'm planning on using VSCodium + continue, but I'm still learning my way through it.

Reply

[-]

mysticzoom@reddit

beyond expectations honestly.

Reply

[-]

trialbuterror@reddit

So wat ur recommend

Reply

[-]

bgravato@reddit (OP)

I'm not recommending anything... I was just trying Qwen3.6 on a "real world" problem I had to solve and it failed. Then I gave Gemma4 a shot, more out of curiosity, to see how different its answer would be and the Dense model was actually able to solve it correctly. This was one specific and isolated scenario, so not enough to draw any conclusions, but I decided to share and see if other people had similar experiences...

Reply

[-]

nunodonato@reddit

My qwen 27B nails everything I throw at it. You must be doing something wrong. I am FP8 but I dont think that would be the cause of the difference

Reply

[-]

bgravato@reddit (OP)

Maybe I am, but I used the exact same prompt in all cases and Gemma4 31b dense did better and in around a third of the thinking time that gwen3.6 27b took... I'll test it again soon with different prompts for different situations.

Reply

[-]

GrungeWerX@reddit

yeah yeah yeah...gemma 4 is so amazing...blah blah blah Until one actually tries to do any *real* agentic work. Then we find out the truth. I don't even use Gemma 4 anymore because it's so d@mn slow at high context that I can't get it to do anything. 26B's intelligence is unacceptable. Does it have a nice voice? **Sometimes**. It's **more warm** than Qwen 3.5, for sure. It's probably "great" for writing assignments or translation for some people. I keep it because I can let it run in the background when I'm not doing any real work and I want to test it out on some revisions. But it's still got a LOT of slop in it. It's not just this, it's that garbage. And other AI-isms. Noticeably better than gemma 3 27B, but still requires some hand holding. Thankfully I don't use AIs to write - they are just for story analysis and lore summarization when I'm doing research.

Reply

[-]

bgravato@reddit (OP)

That tone was totally unnecessary... It was not at all my intention to say Gemma4 is better than hot chocolate and that Qwen sucks... I was actually expecting qwen to deliver, but I was surprised it messed up. I gave Gemma4 a try just for curiosity and I was surprised that it got better results in far less time than qwen3.6 took to give me a wrong one...

Reply

[-]

MrPecunius@reddit

Qwen3.5/3.6 27b & 35b a3b are all very good at generating complex queries for PostgreSQL. Better than me, in fact, and I have been working with pg for over 25 years. I don't see any reason it would be different for MySQL/MariaDB.

Reply