Gemma 4 Tool Calling
Posted by juicy_lucy99@reddit | LocalLLaMA | View on Reddit | 21 comments
So I am using gemma-4-31b-it for testing purpose through OpenRouter for my agentic tooling app that has a decent tools available. So far correct tool calling rate is satisfactory, but what I have seen that it sometimes stuck in tool calling, and generates the response slow.
Comparatively, gpt-oss-120B (which is running on prod) calls tool fast and response is very fast, and we are using through groq. The issue with gpt is that sometimes it hallucinates a lot when generating code or tool calling specifically.
So, slow response is due to using OpenRouter or generally gemma-4 stucks or is slow?
Our main goal is to reduce dependency from gpt and use it only for generating answers. TIA
Voxandr@reddit
on selfhosting it dosent' work properly at all.
EffectiveCeilingFan@reddit
Why is this getting downvoted? While it’s at least “working” now, fixes are still coming in for Gemma 4 daily on llama.cpp. I’d hardly call that working properly. Commenter is completely right.
Voxandr@reddit
There are many US Good / China Bad , Google Fan Bois why the best open source model we have is qwen and GLM. NONE of American models tha twas open source come close to that. They would down vote whoever say bad about Gemma.
false79@reddit
What's your problem? What did you try where it doesn't work?
So far tool calling has been as good as gpt-oss imo.
Voxandr@reddit
false79@reddit
I've had issues with kanban sytyle agent tools. I fell back on to pure CLI.
Apparently, those agentic tooling is hitting a different endpoint than the one in the CLI experience where I've found the tooling more reliable (e.g. cline --tui).
I'm guessing what you are using is open source, so your YMMV, when it will handle gemma 4 tool calling.
Voxandr@reddit
its cline , dosen't matter what ui is (TUI / VSCODE / KANBAN) the same result.
false79@reddit
Yeah Cline Kanban doesn't work and it's in beta. It only works with cloud models to my knowledge. This isn't gemma's fault.
For cline -- tui though, I can confirm on llama.cpp -b8683 that it works with the following:
gemma-4-26B-A4B-it-UD-Q4_K_S
gemma-4-31B-it-UD-Q4_K_XL
gemma-4-E4B-it-BF16 (Not recommended)
Voxandr@reddit
I had tested latest UD Quants (updated 5 hrs ago) and its working better!
Voxandr@reddit
Important_Quote_1180@reddit
Been using the 31b q4 heretic on my 3090 and getting 35 toks gen. Tool calling is great with my Obsidian Vault.
putrasherni@reddit
using it for coding is a dead end , which one are you using ?
bcdr1037@reddit
I've been seeing people mentioning obsidian many times. How do you use that in your day to day work ? Conceptually is it some sort of local notebooklm ?
Important_Quote_1180@reddit
It’s a wiki for your files. It has tags and links to related pages. It’s a very easy to use RAG system for agents too. I can find files quickly because it uses a flat file structure for everything.
bcdr1037@reddit
Thanks!
Important_Quote_1180@reddit
You are most welcome. I’d be lost if not for Reddit comments
teachersecret@reddit
31b is a dense model, so it's going to be a bit slow. OSS-120b is 'bigger', but it activates a far smaller piece of the model and is rather quick.
If you wanted speed you'd have to drop down to the 26ba4b model which might not get your job done.
dylantestaccount@reddit
Gemma 4 31B is just incredibly slow on all providers on OpenRouter. The fastest is Venice offering 32tps throughput. The average is like 20.
juicy_lucy99@reddit (OP)
That's what I am thinking too in the case of OpenRouter. I am thinking to give advice to my client to deploy it own, it will be much faster. Also I did notice that gemma-4-26b-a4b-it was much faster than 31B on open router.
iits-Shaz@reddit
The slowness you're seeing is almost certainly OpenRouter, not Gemma 4 itself. OpenRouter adds routing overhead and you're at the mercy of whatever backend provider they assign you. Gemma 4 on dedicated hardware is fast — I'm getting 30 tok/s generation on the 2B model running locally on a phone, and the person above is getting 35 tok/s on the 31b q4 on a 3090.
The "stuck in tool calling" issue — I've seen this too. Two things that help:
Limit the number of tools per invocation. If you're passing dozens of tools, the model spends more tokens reasoning about which one to pick. I score tools against the user query (BM25 ranking on tool names + descriptions) and only pass the top 5-6 to the model. Massive improvement in both speed and accuracy.
Set a max chain depth. If the model calls a tool, gets a result, calls another tool, gets a result... you need a hard cap (I use 5). Without it, the model can loop — call tool A, not like the result, call tool A again with slightly different params, repeat forever. That's probably your "stuck" behavior.
For the hallucination issue with GPT on tool calls specifically — structured output mode (force JSON schema) helps a lot. If the model can only output valid tool call shapes, the failure mode shifts from "hallucinated tool name" to "wrong parameters," which is way easier to catch and retry.
If your goal is reducing GPT dependency, running Gemma 4 locally (even the 12b) on a decent GPU would eliminate both the OpenRouter latency and the API cost. The tool calling fidelity on Gemma 4 is genuinely good — the issue is your serving layer, not the model.
EffectiveCeilingFan@reddit
There is no Gemma 4 12B :P, hi ChatGPT!