Yes and no.

It is pretty well-established now that an LLM's skillset is determined by the comphrehensiveness of those skills' representation in its training data, and its competence is determined by the quality of that training data and the model's parameter count.

Trainers are thus able to pick and choose which skills a model exhibits, and each training organization has their own priorities. IBM's Granite models, for example, have a fairly sparse skill set, and those skills are fairly specific to business applications. The further implication is that as training datasets become increasingly exclusive of low-priority skills and subject matter, it will be up to the open source community to identify gaps in frontier models' skills and topics, amass training datasets which fill those gaps, and amend models with further training without causing catastrophic forgetting.

High quality training data is still a tricky wicket. Synthetic datasets help, and so does reward-model driven curation, but those are both very compute-intensive, and training data curation still requires the attention and labor of SMEs, who are in limited supply, in high demand, and expensive to employ.

It seems pretty clear that inference quality increases only logarithmically with parameter count, which hits the point of diminishing returns pretty quickly, but we are still learning new ways to make best use of a given parameter budget. There was a recent paper, for example, demonstrating that as the ratio of training to parameters increases, parameters encoding memorized knowledge get cannibalized to encode more generalization capabilities. That will have a profound effect on how we train and evaluate models, but I think it may take a while for the implications to seep outward to the largest players.

There is also still some low-hanging fruit to be plucked at the other end, at inference time, where we can utilize more resources to increase the effective skill sets and competence of existing models. "Thinking" is one example of this (which does not require thinking models, but can be emulated with most models via multi-pass inference), but we can also improve inference quality by means of self-critique, self-mixing, RAG, and more sophisticated forms of Guided Generation.

I think you are right, that there is a lot of optimization to do, too, but there is no shortage of other improvements to keep us busy.