Using it on my low-mid range Android phone, I'm extremely happy with it and I'm very impressed. I've only done writing / roleplay things and random question-asking with it so far, but considering the size it's very solid.

A huge benefit is that, at least on my phone, it's very fast comparatively. In the same circumstances (Same token character card, same first message and first user reply totaling around 600 tokens, cleared chat history and cache beforehand), here's the time before it starts generating text:

• 4_K_S Phi-3 : 128 seconds

• 4_K_M Gemma 2 : 56 seconds

And the token generation itself is much faster too; I don't have official numbers, but eyeballing it I'd say it's more than twice as fast too. Since follow-up replies starts much faster than the initial load and the token generation is close to what I can read, it's actually fast enough for me to use "live" instead of inputting something and look at it later. Some other LLM like StableLM 3b isn't much slower and StableLM 1_6b is slightly faster, but the quality of the output definitively doesn't reach the level of Gemma 2 even with their best finetunes like Zephyr.

For random question-asking (for example, questions about the purpose and benefits of certain Olympics equipment and outfits), it's not hugely knowledgeable but it still gave the best answers of any models I tried Phi-3 size and down. It missed some key reasons, but the ones it gave were still correct and the formatting was superb, with headers, bolds, bullet points, etc.

For writing, it's the only small model that can handle my "Choose your own Adventure" card correctly; it always offers options A- B- C- D- as requested on every prompt, and actually follows the letter I pick. All the other small models either don't give options, or give correct formatting but don't actually follow what I picked (StableLM finetunes all seemingly do that) and generate a new floor every time. Even Mistral 7b 0.2 finetunes tend to pick a letter to follow by itself. Gemma 2? Works perfectly.

For roleplaying, it's also very good. I have a card that has two different characters, and it can actually handle that fine. From what I tried so far, Gemma 2 also never wrote as my character either with a few very different cards: in my experience that's a common issue with any models type smaller than Llama 3 (Phi-3 being hit-and-miss, and other models, even including Mistral 7b being mostly miss). Gemma usually follows the card's initial message and first user message format too: If it's made in a "It refers to the user as You" way, it usually does that, and if it's "It refers to the user as Ayra", it follows that too. Same for how the characters refers to themselves; in most small models, it usually flips around randomly between the two. Gemma 2 writing style is also pretty decent and interesting to read; it's less flat than Phi-3's in my opinion.

It's definitively not perfect: spacial awareness seems quite poor for example. And I'm definitively not claiming it's beating Llama-3 8b, especially not the better writing-focused finetunes like Stheno 3.2. And even less bigger models that I've never tried, like those 70b ones. I doubt it's going to to impress anyone who is used to roleplay with bigger models.

But for something that can run on my mid-range phone at very acceptable speeds? It's stupidly good. The output is much better than other smaller model, with Phi-3 being the only real competition. For me, Gemma 2 runs over twice as fast as Phi-3 (and makes the app crash less), and I certainly prefer how Gemma 2 writes.

I've been spending more time testing and looking for "the LLM that does what I want on my phone with acceptable speed" than I have been actually using them... But finally, with Gemma 2 I feel like it's a "This is what I've been looking for!" moment. Fast enough, rational enough, writes well enough. I'm thrilled!