This is not too surprising.

LLM competence increases only logarithmically with parameter count, and training hardware requirements increase by the square of parameter count (rule of thumb is FLOPs = 6 x P x T, where P = parameter count, T = training tokens, and training tokens is conventionally proportional to parameter count, by a factor of anywhere from 100 to 800. Thus FLOPs = 6 x P^2 x r where r = that ratio of training tokens to parameters).

Thus, training compute resources increases per the cube of competence improvement.

If Moore's Law still held up, it would be expected to outpace compute requirements eventually, since O( k^N ) increases faster as N grows large than O( N^k ) for all k > 1, but Moore's Law hasn't been well for nearly ten years now.

LLM tech is nifty and all, but this shit just isn't sustainable.