Which Gemma model do you want next?

Yea, I am curious how much more it could be scaled as well (and how much more they could get out of it, if they did scale it significantly), like if all it could do is mainly just increase the vocabulary size more and more, but not much else, or if it could be used in more interesting ways than just that. I was going to link [this thread](https://www.reddit.com/r/LocalLLaMA/comments/1sd5utm/perlayer_embeddings_a_simple_explanation_of_the/) that -p-e-w- made about it a couple weeks ago that has a lot of in-depth explanation about it, but I was browsing through it and I see you were one of the commenters in it so you already saw it. Most of it is too advanced for me to understand yet, so I don't really know if it can only be used for vocab (and thus not do anything too crazy as far as scaling it massively) or if it can be used for all sorts of other aspects of an LLM, and thus become the future of LLMs in some big way. If it can be used the latter way, then I'd think it would already be a main architecture of model for big models though, no? Like why aren't they already scaling it like that and using it that way? Or is Google the only ones that know how to do it so far, and have just chosen to only do it at small scale to tease us with it a bit, and others would do it at big scale and do crazy shit with it, but they just don't know how yet? Anyway, pretty interesting. Kind of makes me want to learn more in depth stuff about how LLMs work to see if I can understand it better