scaling is dead

The real scaling problems are far more practical. You can't iterate quickly and build new products when you need to build out an entire new data center to improve your models. Shrinking models enough that they can run on commodity-ish hardware allows teams to iterate much faster on new architecture designs, which are where the real gains will come from. We've already seen the risks of model collapse, which only became apparent in 2023 once we had already exhausted the entire internet for training data. I don't expect that scaling to nuclear powered data centers will give as much improvement as you expect. We don't really have more data to throw at the problem, and adding more parameters to our models isn't scaling like it used to. Besides, the 80/20 rule applies. If an AI can give 80% of the performance for 20% of the training costs, they're probably going to have a better product than their competition because they can build 5 models for the cost of their competition. Plus, if we can shrink the current frontier models by 80%, it starts to become reasonable to run them on hardware normal people can buy. If we scale RAM production so consumer hardware can have >100gb of RAM (only a 4x increase from 32GB), then a 80/20 frontier model can run in places without stable internet access, places where security or reliability are critical. A local frontier model wouldn't deal with unstable latency issues or data center scaling problems, making it a far more compelling product to many businesses. Edge computing is very powerful, but that only works if models can run on a moderately sized cluster instead of needing an entire server rack.