Been testing gemma 4 since google dropped it. the small variants E2B and E4B are genuinely impressive on device. 40+ tps on iphone with mlx optimization, 128k context window, handles image and audio natively. feels like magic for basic chat and quick questions.

Ran it on my m5 pro macbook too. the 26B MoE version is fast for direct conversation. text generation, code explanation, summarization all smooth.

Then i tried using it as an actual coding agent and everything fell apart.

The problem isnt raw intelligence. its tool calling and structured output. agent workflows need the model to reliably call functions, parse results, chain multiple steps together. gemma 4 keeps choking on this. outputs malformed json, misses required fields, gets confused mid-chain. tried it with aider and it would stall, throw errors, or produce structurally broken responses.

Switched to qwen3-coder in the same setup. same framework, same tasks. file creation, command execution, multi step refactoring. all worked fine. the difference isnt general capability, its whether the model was specifically trained for agentic tool use patterns.

This is the gap nobody talks about when they get excited about on-device models. running a model locally is one thing. running it as a reliable agent that can plan, execute, verify, and iterate is completely different. the agent loop requires consistent structured output across dozens of tool calls. one malformed response breaks the whole chain.

For simple stuff gemma 4 on device is genuinely useful. quick code explanations, reviewing a function, answering questions about syntax. zero latency, zero cost, works offline. great for that.

But for actual development work where you need the model to autonomously write code, run tests, fix failures, and iterate? cloud models are still way ahead. the reliability gap for agentic workflows is massive.

The business model implications are interesting though. if on-device models keep improving, they eat into the high-frequency simple query market. cloud providers will have to justify their pricing with capabilities local models cant match. complex multi-agent orchestration, massive context windows, reliable tool calling chains.

Tools like verdent and cursor that run multi-agent workflows with verification loops are exactly the kind of thing that needs cloud-grade models. you cant have an agent that fails 1 in 5 tool calls when its running a 20-step automated pipeline. the compound failure rate kills you.

Short term: local models for quick stuff, cloud for serious agent work. long term: depends on how fast on-device tool calling reliability improves. but were not close yet.