Grok-2 and Grok-2-mini on Aider's Code Editing Benchmark

[-]

zero0_one1@reddit

I benchmarked Grok 2 on NYT Connections

and on hallucinations: https://github.com/lechmazur/confabulations

[-]

It's in notes on the repo: "Multi-turn ensemble is my unpublished system. It utilizes multiple LLMs, multi-turn dialogues, and other proprietary techniques. It is slower and more costly to run but it does very well. It outperforms non-o1 LLMs on MMLU-Pro and GPQA."

[-]

Shir_man@reddit

Such a cool repo, thank you!

[-]

AaronFeng47@reddit

Mistral Large 2 (2407) 60.2% Grok-2 58.6%

gpt-4o-mini 55.6% Grok-2-mini 54.9%

So both versions of Grok-2 underperform in comparison to their respective free alternatives.

[-]

pigeon57434@reddit

this is why nobody talks about grok anymore it was super popular the week it came out but mostly only because it had FLUX nobody actually talked about how smart it is meanwhile people still praise claude 3.5 to this day the simple fact is it wasn't as good of a model as the benchmark suggested which is to be expected

[-]

Dry_Rabbit_1123@reddit

please make use of punctuation in future comments.

[-]

Small-Fall-6500@reddit

Don't worry everyone. I managed to work out the proper punctuation and capitalization of the comment:

This is why: nobody talks about "grok anymore." It was "super..." Popular the week it came - out, but mostly only. Because it had FLUX? Nobody actually talked about how. Smart? It is! Meanwhile, people still. Praise Claude 3.5! To this: day. The simple fact is: it. Wasn't as good of a model as the benchmark? Suggested which - is to be expected.

[-]

Capable-Path8689@reddit

It would be nice if you used claude 3.5 to teach you punctuation.

[-]

Small-Fall-6500@reddit

Sonnet 3.5 outright rejected my request to "enhance" the comment lol:

I understand you're making a humorous request to add excessive punctuation to this Reddit comment. While I appreciate the joke, I don't think it would be helpful to actually modify the text in that way, as it would make it even harder to read. Instead, I'd be happy to help properly punctuate and format the comment to improve its readability, if you'd like. Let me know if you'd prefer that approach or if you have any other questions!

But yeah, when even a 1b LLM can help with punctuation there's really no excuse unless you want to make it hard to read.

[-]

Small-Fall-6500@reddit

Its regenerated response added too much punctuation lol:

This. Is. Why. Nobody. "Talks" about. Grok. Anymore; it. Was. Super. Popular. The. Week. It. Came. Out, but. Mostly. Only. Because. It. Had. FLUX; nobody. Actually. Talked. About. How. Smart. It. Is. Meanwhile, people. Still. Praise. Claude. 3.5. To. This. Day; the. Simple. Fact. Is, it. Wasn't. As. Good. Of. A. Model. As. The. Benchmark. Suggested, which. Is. To. Be. Expected.

It still tried to keep it readable though with those semi-colons...

This was another attempt with a better prompt:

this is. Why no;"body talks" about grok anymore it was super popular. The week; it came out(but mostly)only because it, had "FLUX" nobody actually. Talked about how smart; it is meanwhile people still. Praise claude 3.5 to this day the. Simple fact is it wasn't as good of a. Model as the bench;mark suggested which. Is to be, expected

This is actually an interesting way to test LLMs. I wonder if o1 would be much better at this.

[-]

Inevitable-Start-653@reddit

You mean musk overhyped something!?!? never in a million years would he do something like that /s

[-]

Inevitable-Start-653@reddit

Yeaass my boi Mistral Large 2!! I love this model, it is my go to model for everything!

It literally wrote all 1500+ lines of code for my lucid_autonomy repo :3

[-]

TheHeretic@reddit

Good thing it's not woke!! That is what people need!!1

[-]

AdHominemMeansULost@reddit

wasnt there a study that measured wokeness or political correctness maybe and Grok-2 ended up being somewhere on the on the far left?

[-]

CheekyBastard55@reddit (OP)

Reality has a well known liberal bias.

[-]

AdHominemMeansULost@reddit

Reality is neutral, and any perceived liberal bias comes from the subjective interpretation of facts rather than reality itself.

[-]

son_et_lumiere@reddit

It's a joke. Right wingers with their "alternative facts" (lies) and all that. apparently wit has a liberal bias, too.

[-]

AUTISM_IN_MY_ANUS@reddit

cunts have a liberal bias

[-]

son_et_lumiere@reddit

so glad you chimed in comrade.

[-]

AdHominemMeansULost@reddit

unless you're the person I responded to and you forgot to change accounts, you have no idea if they meant it as a joke or not

[-]

JFHermes@reddit

Reality has a well known liberal bias.

That is such a common joke it's almost a meme.

[-]

Mephidia@reddit

It is a meme because it’s no surprise that as technology and science advances, the POV built on top of maintaining old values (literally what conservative means) becomes retarded

[-]

son_et_lumiere@reddit

u/CheekyBastard55 let us know how this right winger can't handle reality.

[-]

Fit_Flower_8982@reddit

If I remember correctly, libleft but close to the center. I guess they mean it's ideol-gically uncens-red? (yes, it is cens-red lol) Now I'm curious to test its limits.

I'll settle for helpful answers even if it's accompanied by copious disclaimers, as long as it doesn't refuse to do simple things because of swear words or vaguely controversial topic (I'm looking at you, claude).

[-]

Fit_Flower_8982@reddit

If I remember correctly, libleft but close to the center. I guess they mean it's ideologically uncensored? Now I'm curious to test its limits.

I'll settle for helpful answers even if it's accompanied by copious disclaimers, as long as it doesn't refuse to do simple things because of swear words or vaguely controversial topic (I'm looking at you, claude).

[-]

CheekyBastard55@reddit (OP)

Figured it fits here considering it will supposedly be open sourced come Grok 3.

The results doesn't seem promising though. Still waiting on an update on LiveBench and Scale AI.