I did not miss your point. How you view intelligence is fundamentally different.

IQ Tests are carefully tuned, validated, and hardly ever change.

Sure no argument there.

SATs and similar tests are written by lay people. They are not written by experts.

I do not know how the SAT tests are written or administered, the O and A levels I took were written by professors in Cambridge Universities, which I consider subject matter experts

Exams are specifically meant to be able to be passed.

Again I don't know about SAT, but O and A levels are designed to assess a population of students on their aptitude in academics via a bell curve, the Gaokao especially is extremely brutal from what I hear. I know people who have failed their Os and As.

It has seen the IQ tests before.

Sure, if it was fed IQ test answers as training data, it will just regurgitate said data, not impressive, agreed.

But these national exams and tests are always worded at least slightly differently. LLMs being able to apply the same concepts (if trained on) applying it to different scenarios, which is expected of from students, which dictates their further education opportunities and hence early career, and do it with such speed from inference, it should raise an eyebrow.

Another way to think of this is that anyone with enough patients and a search engine could pass nearly any exam.

This is also what is expected of the test takers if they are not efficient in learning and want to score well isn't it.

In subjects for which I am an expert I can easily find ways that it makes errors, particularly when presented with an novel argument. LLMs have a massive propensity to analogize, incorrectly, to arguments they are familiar with (ie that was in their training set).

This is not what they were created for. They are general models. There is simply no market right now for specialised models outside of perhaps coding models, and maybe in the open source free market RP and creative writing models. You have had decades of specialisation and focussed learning and experience, LLMs have technically existed for less than 1.

More broadly what they excel at is, a higher than average baseline knowledge of an almost unlimited areas of expertise, I am more than willing to bet the SOTA models will destroy you in a matter of knowledge tests of a few thousand topics, you have never even heard of before, same goes for me and anyone else in the world, we just physically do not have the time to be able to learn as much as an LLM in as many topics, and be as cheap as inference compute. That is my idea of it being smarter than humans.

"A jack of all trades is a master of none, but oftentimes better than a master of one"