yeah, no surprise to a lot of you here. it was llama.cpp (thanks u/jacek2023) and my faffing about trying to identify and fix bugs in the gguf were pretty much pointless in the end (except i learned some useful shit i guess). for anyone who cares here's my story this morning.

setup: win11, rtx pro 6000 96gb (blackwell), lm studio serving gemma-4-31b-it Q4_K_M to opencode and qwen code agent harnesses. comparing against qwen3.5-27b which has worked great for tool calling. gemma 4 would get stuck in infinite tool-call loops. completely unusable for agentic work despite google's benchmark claims.

tl;dr

the problem was (as others have already pointed out) lm studio's bundled llama.cpp lacking the gemma 4 specialized parser (PRs #21326, #21327, #21343, #21418). the gguf metadata does seem to have real issues too (missing eog_token_ids, wrong token types on tool-call delimiters), but the current llama.cpp runtime compensates for those automatically. so, woops. i'm clearly a novice here.

the fix: use llama.cpp b8664 or later with --jinja. that's it. grab the pre-built release from github, point it at the stock gguf, done. no gguf patching needed.

and, yeah, benchmarks aren't lying. gemma 4 genuinely is good at tool calling. but "good at tool calling" and "works in your local agent stack today" are different claims, and the gap between them was a handful of missing parser code in the runtime.

if you're on lm studio, sit tight until they update their bundled llama.cpp. or just run llama-server alongside it on a different port.

the whole story

step 1: the a/b curl tests (isolating the failure)

before touching anything, we wanted to prove where the failure actually was. ran identical curl tests against lm studio's openai-compatible endpoint for both models.

test 1 — single tool call (weather tool): both models passed. clean finish_reason: "tool_calls", valid json args. gemma was not broken at basic tool invocation.

test 2 — round trip (tool call → tool result → final answer): both models passed again. gemma accepted the tool result, gave a clean natural language answer, stopped properly.

test 3 — nested json schema (create_task with arrays, enums, nested objects): both passed. gemma handled the richer schema fine.

test 4 — multi-step two-tool chain (search_files → open_file): this is where gemma fell apart. lm studio logs started spamming:

Start to generate a tool call...
Model generated a tool call.
Start to generate a tool call...
Model generated a tool call.

over and over until ctrl-c. qwen completed the same test cleanly. so the failure was specifically in multi-step tool sequencing, not basic tool calling.

step 2: gguf metadata inspection (the red herring that taught me something)

vibed a raw binary parser (no dependencies) to inspect the gguf header. found a few possible problems:

one: tokenizer.ggml.eog_token_ids: completely missing. this is the list that tells llama.cpp when to stop generating. without it, the runtime only knows about EOS (token 1, <eos>). but in multi-step tool flows, <turn|> (token 106) also needs to be recognized as a generation stop point.

two: tool-call delimiter tokens typed wrong:

• [48] <|tool_call> — USER_DEFINED (4) instead of CONTROL (3)
• [49] <tool_call|> — USER_DEFINED (4) instead of CONTROL (3)
• [50] <|tool_response> — USER_DEFINED (4) instead of CONTROL (3)
• [51] <tool_response|> — USER_DEFINED (4) instead of CONTROL (3)

three: meanwhile <|tool> (46) and <tool|> (47) were correctly CONTROL. someone missed the inner four during conversion.

four: token 212 </s> typed as NORMAL (1) — this is the one lm studio warns about on load. it's actually an html tag in gemma's vocab (not the real eos), but lm studio gets confused because </s> traditionally means eos in other models.

vibed up a python script that patched the gguf: fixed the token types, added eog_token_ids = [1, 106], rewrote the header and copied \~18gb of tensor data. total size difference: 64 bytes.

result: womp womp. still looped in lm studio. the metadata seemed like real bugs but not the root cause of the looping. and maybe i'm just completely wrong about this.

in any case, this is where u/jacek2023's post pointing at the llama.cpp PRs became the key lead.

step 3: the actual fix — llama.cpp runtime

gemma 4 uses a non-standard tool-call format:

<|tool_call>call:function_name{key:value,key:value}<tool_call|>

with <|"|> for string quoting instead of standard json. every layer of the stack needed new code to handle it, and those fixes literally landed a couple days ago:

• PR #21326 (apr 2) — gemma 4 template parser fixes, added normalize_gemma4_to_json() and a dedicated PEG parser
• PR #21327 (apr 2) — tool call type detection for nullable/enum schemas
• PR #21343 (apr 3) — tokenizer bug where \n\n gets split into two \n tokens, causing garbage in longer sessions
• PR #21418 (apr 4) — gemma 4 specialized parser

as others have pointed out, lm studio bundles its own llama.cpp and hadn't pulled any of these yet.

grabbed the official pre-built release from github (b8664, released same day; windows binaries with cuda 13.1 for blackwell). no custom build needed, just a folder of exe + dll files.

launched with:

llama-server.exe ^
  --model gemma-4-31B-it-Q4_K_M.gguf ^
  --host 0.0.0.0 --port 8090 ^
  --n-gpu-layers 60 --ctx-size 262144 ^
  --threads 12 --batch-size 512 --parallel 4 ^
  --flash-attn on ^
  --cache-type-k q8_0 --cache-type-v q8_0 ^
  --mlock --jinja

the --jinja flag tells llama-server to use the model's own chat template instead of a hardcoded one, which i guess is required for gemma 4's non-standard tool format.

step 4: the payoff

re-ran the exact multi-step two-tool test on my patched gguf that caused infinite loops in lm studio:

no looping. no repeated tool-call generation. model even included coherent reasoning about which search result was the best match.

then pointed both opencode and qwen code at the llama.cpp endpoint. both are working beautifully now. multi-step tool chains, file reading, bash execution, the whole deal. gemma 4 even successfully adopted my custom agent persona, made jokes, and self-validated its own model by curling its own endpoint. all the stuff that was completely broken before.

step 5: controlled experiment — do the gguf patches even matter? nope lol

this bugged me. changed two things at once (gguf metadata + runtime) and didn't know which one was actually load-bearing. so loaded both the original unpatched gguf AND the patched gguf side by side on llama.cpp b8664 (different ports, same machine, 96gb vram makes this easy) and ran identical tests against both.

the original unpatched gguf worked perfectly on b8664. identical behavior across all three steps. the runtime auto-infers the eog tokens and overrides the wrong token types on its own — you can see it in the load logs:

control-looking token: 212 '</s>' was not control-type; this is probably a bug in the model. its type will be overridden

good to know! i've not really understood these stacks at this level. boo on me.

so: you don't need to patch the gguf. the metadata issues might be real bugs in the file, but llama.cpp b8664 compensates for all of them at runtime. yay.

been testing some more complex agentic stuff in both opencode and qwen code and so far the model is killing it. i'm happy now. 🙌

noticed the issues you're describing using lm studio + opencode. we did a pretty minimal repro on lm studio's openai-compatible endpoint with curl, using the same prompts/tools for qwen3.5-27b and gemma-4-31b-it@q4_k_m.

we found that both models handled the simple case fine. single tool call worked, both also handled the simple round-trip fine (tool call -> tool result -> final answer), both also handled a harder nested json tool schema fine.

so at first it looked like gemma was innocent, but then we tested a tiny multi-step agent flow with 2 tools: search_files, open_file

prompt was basically "find the file most likely related to lm studio tool-call failures, then open it."

qwen behaved normally. first call search_files, second call after fake search results open_file, no weirdness.

but sweat sweat gemma is where it got ugly. on the multi-step flow, lm studio logs started spamming start to generate a tool call... and model generated a tool call.

over and over and over until i came in with a ctrl-c hammer. so yeah, gemma + lm studio/llama.cpp def falls apart once the workflow becomes multi-step/agentic. bummer.

seems pretty consistent with what people in this thread are describing where toy setups seem to work, but more realistic agent/tool workflows break. and parser/template/runtime issues seem like the culprit. which, we've been through all this before.

also worth mentioning. i'm seeing lm studio logging some sketchy tokenizer/control-token stuff on gemma load (this is probably a bug in the model. its type will be overridden, the tokenizer config may be incorrect.

qwen3.5 serious is just way more stable for this use case right now. it's actually useful in the opencode harness. gemma 4 just isn't right now.

if useful i can post the exact curl, but the short version is basic function calling passed, multi-step tool sequencing is where gemma eats shit.