Gemini Pro is why measuring intelligence is hard.

Posted by Sicarius_The_First@reddit | LocalLLaMA | View on Reddit | 13 comments

OK, so I really hate Gemini 2.5 \\ 3.0 pro with a passion, but the amount of knowledge it has is second to none. I saw some benchmarks that shows it's the best coding LLM in the world, however this is BS, either trained on benchmarks, or by accident contaminated training data, because in real world usage Claude wipes the floor with Gemini (all versions, and while this is my opinion, it is shared among many people I know). It's like an autistic savant with zero common sense, and an insane amount of knowledge. About the knowledge, everyone knows that Gemini (and gemma models too) got an absurd amount of obscure knowledge. It will know everything there is to know about some unimportant anime character from an obscured anime no one knows, that was only in 1 episode. Also,I made an accurate (to the best of my knowledge) photo realistic photograph of the Jurassic period, cropped it so no dinosaurs and only vegetation was visible, nothing else. https://preview.redd.it/934s8y30rphg1.png?width=1448&format=png&auto=webp&s=f901c48841c2f18488be16a1fb336b08028cf32c Gemini accurately determined that this was a photo of a Jurassic period, no hedge. Very impressive. We are at a point where knowledge does not worth smart model, it's something that ironically is very human. If we had the common sense & humanity of Claude, combined with Google's dark voodoo of knowledge graphs with all of the human knowledge (which they have), idk about AGI but it would sure as hell be close to it. Whatever AGI even means at this point. What do u guys think? Gemini or Claude?

13 Comments

[-]

FPham@reddit

IDK, these "wipe the floor" statements are just horribly inaccurate. Claude is a super stable coder and code fixer, but often it's first algorithm generation (especially novel or nontypical algorithms) is all over the place. But funny enough Gemini -3 can devise a new well structured algorithm (that is logically sound, but of course doesn't work) then Opus will be "this is a fantastic code with a small detail to be fixed". Then you go back and forth and have a chance to actually have something new and working. Karen 5.2 (I mean ChatGPT) is at the weird position of making stuff up all the time when it is outside it's expertise - basically agreeing with whatever nonsense you ask even if it has nothing to do with solving issue. "This line is exactly the problem!". OMG, trying to make my 2x3090 work properly in Axolotl with the help of ChatGPT is a constant fantasizing about new option parameters I should add to yml that never ever existed. It's like 2022 again!

No_Afternoon_4260@reddit

Funny to think that both google and anthropic are Alphabet 🤔 2 faces of the same coin.

my_name_isnt_clever@reddit

This is just straight up incorrect. Anthropic is independent.

Indeed I'm sorry I always thought that. Google owns 15% of anthropic

zball_@reddit

GPT pro is the real answer if you tell about "intelligence", or the ability to get work done confidentially. I'm yet to see a case beside writing/editing work that Opus really outperform gpt pro/xhigh. And writing-wise Opus is not the best either,

Eyshield21@reddit

agree that single-number “intelligence” scores miss the point. models trade off knowledge breadth, reasoning/consistency, and instruction-following. Gemini often feels encyclopedic but brittle; Claude tends to be more reliable in long-form reasoning. for practical comparison i’d use task-specific evals (your actual coding/debugging workflows) and track failure modes. if you share your top 2–3 tasks, people here can recommend the best model for each.

Sicarius_The_First@reddit (OP)

I'm just using Claude when I need complex stuff to be done, Gemini was just on this 5$ a month for Pro sale, but I'm not even using it that much, it was a mistake to get it. Claude is expensive as hell, but it is actually good.

7657786425658907653@reddit

Gemini routinely 'streamlines' my code by deleting half of the app. claude having the code to the right and updating it as you go is so much better.

Agreed, also one of my issues with Gemini is that it always assumes it is correct, and that the user is wrong. It's super hard to convince Gemini to correct course.

CodProfessional3712@reddit

It's kind of a shame because the Claude models are so costly. You lose some, you win some, I guess. Does make me wonder about designing a system that can properly and automatically route simple sub-tasks to a local fast model, which should take the bulk of the costs away for \*most\* tasks.

rolyantrauts@reddit

Worst thing about Gemini pro, is its like an other enthusiastic intern who constantly adds things 'to help'! I am often screaming why did you change that, please just do as I say! Its also very strange as its not gradual it will just get a huge amount of amnesia and change large fundamental parts. But if you get it prompted correct in a short number of prompts boy its amazing what a good analyst it is than just coder. Also sometimes it argues with you that you are wrong until you get quite 'Do as I say for F sake!!!', which I kind of like in a perverse way :)

SlowFail2433@reddit

Yes there are benchmarks that test internal world knowledge without searching such as HLE-no-tools and AA-Omniscience bench and Gemini does do best on these

scottgal2@reddit

No LLM can ever reach animal like intelligence (or AGI, useless term though that has become). They lack the executive function, don't learn post training, don't dream, have no real agency (they exist only in interactions at present) etc...This renders any discussion of AGI / animal like intelligence pointless - because of that fundamental limitation. Calling it 'autistic' is both insulting and lacking in knowledge of both LLMs and neuroanatomy.

Reply to Post

13 Comments