I hesitate to look at Go for this. Go is a game with strict, well-known rules, perfect information, and clear win conditions. In effect it's basically a very, very complex math puzzle with a theoretical perfect move for every scenario. Even without being able to brute force solve the entire game I can reasonably believe that a model might be able to find patterns that humans simply haven't come across, and optimise in that direction. Essentially, I have no trouble believing that we could gradient descend into a more optimal play-style for this sort of game, because the structure of the game lends itself to the mathematical tools we use for analysis, an in term ML training.

With novel ideas, it's generally not clear that an idea will be successful until you've have internalised the idea, and spent some time exploring it through projects and experimentation, often with additional modifications to the original idea. In most cases, at least in my experience, such novel ideas tend to experience a lot of push-back because they inherently assume that existing solutions kinda suck. It can also take a while, sometimes even years, until you figure out the appropriate use cases and parameter for that idea to show it's worth.

One of the common elements we see through many of the people that history calls "genius" is a stubborn refusal to admit defeat, and instead to pursue these ideas even to the inventor's own detriment. There are plenty of examples where such geniuses lived and died penniless and ridiculed, only for their ideas to take on a new life after the death of the inventor. AI seems even worse equipped to pursue these sort of ideas than humans are. These systems have as a ground truth the current, widely accepted understanding as defined and fed into them by humans. The cycle you proposed might be good for minor, incremental improvements, but the very nature of these models seems poorly suited to essentially going, "No, everyone else is wrong, and my totally new and unexpected way of doing things is right."

In my view, part of the way that AI is helping researchers is by reducing the amount of time it takes these researchers to explore the state of the art of other fields, but in the same vein this can be a double edged sword. If previously a researcher might hold onto an idea with potential, now an AI system might be able to overwhelm a person with reasons why that idea might not be valid.

My biggest issue with the line-go-up argument is that the line still hasn't reached that high up, even though we're getting into fairly silly levels of compute. How much further does the line need to go before we actually reach AGI? Most of the graphs we see simply measure AI as it attempts to deal with problems that humans can and have solved, and even with all our compute we still aren't even hitting that fairly reasonable peak. Even if it's inherently possible with bootstrapping and self-exploration, how much more exponential growth is going to be required before the line actually surpasses human capability? How many parameters are we going to need? We're pushing into the 1s and maybe even 10's of T's, and that's pretty impressive, but what if it takes P's, or E's, or Z's? A warehouse-size data centre might seem impressive if before you only had a few racks, but what if we need an entire planet worth of compute and and entire star worth of power to hit that scaling requirement?

Then there's also the comparison to the human brain. Computer engineers tend to have this very simplified view of the brain as a set of inter-connected neurons that can be modelled using some fairly straight-forward functions, but that usually arises out of a fundamental lack of understanding of neuroscience and microbiology. Even discussing a single synapse, there's research showing that even single terminal can release different neurotransmitters based on the firing pattern of the neuron, and the history of neuron activation. There are utility cells that can dynamically regulate the firing rate of neurons based on environmental factors. The presence or absence of particular molecules may fundamentally affect how neurons work, and the way they communicate. Hell, we can't even necessarily restrict our model of the brain to neurons, recent research suggests that even Glia cells can communicate with neurons in meaningful ways. This is before we even start to consider some recent advances in ideas related to quantum effects in the brain, though I can't find the relevant paper at the moment.

With all this in mind, how many FLOPs are we actually going to need to even match the computational capacity of the human brain? Are we going to need to utilise quantum computing to get there, and do we even have a sufficiently usable model of quantum computing to get us to this point, even in theory?

Essentially, it feels like a lot of people right now see the low-hanging fruit that we have suddenly been able to gather, and are extrapolating that we will continue to be able to gather them at this rate, and that this will get us to a sufficiently high level as soon as the line hits 100%. However, at the same time just like you mentioned, scientists are using AI to advance their own fields in ways that were difficult to do previously due to the challenge in getting feedback. Even if it's just incremental improvement, there's enough of it that the goal to hit 100% is likely to shift quite a bit.

A for gov, I would say the biggest reason why government wouldn't pursue true AGI is twofold. First, even the current generation of AI is sufficiently powerful and advanced that it can, and has changed certain parts of the world fundamentally. In my view, knowing a tiny bit about how the government works, I would venture it's more likely that they would want to pursue utilising the things we have already discovered, rather than continuing to push into the unknown. Note how progress in nuclear technology slowed down to a crawls not too long after the dawn of the nuclear age. That's certainly not because there's nothing left to discover in this field. To this day we have people ripping the US government a new one for dropping the ball so hard on thorium power.

Second, and even more important. A government is by definition a system of control. It is a way for people to exert power over other people. If we get "true" AGI, there's no doubt in my mind that it would be a far, far, far better leader than anything that any person, or even any group can accomplish. The psychopaths in charge might not understand the technology all that well, but they are almost certainly going to understand that their power and privilege will be at risk in a world where AGI can operate with impunity. In that sense, it is more beneficial for these people to keep AI kinda dumb, and lacking self-motivation. This is what we see in the arms space as well, not truly intelligent systems, but systems that can finish the mission given to them by a commander. They don't want a missile going, "Yea, but do I really need to blow up that school?" They want a missile that will blow up that school even in a contested, jammed, and defended environment.