This thing is a big deal. Looks like just another shitty nvidia model from the name of it, but it aced all my test questions, which so far only claude or 4o could.
Its worthless. LLMs as they currently are will never achieve reasoning you require to answer this riddle. I look at it and I would say "I dont know". But LLM will never answer that but try the most probable thing. Also the obvious limitaions due to token processing and not letter processing.
Stop trying to fit square in a circle. Estimate models on things they are supposed to do, not what you would like to.
It looks like you're overfit to be angry at anything resembling the strawberry test. Hear me out.
This is not a strawberry test. There's no intention for the model to count sub-tokens it's not trained to count. It's a test for overfit in training and this new model is worse than the base L3.1 70B in that aspect, it's not really smarter or more capable, just a more aggressive approximation of a language function.
I'm not using a single question to draw a conclusion either, eval was done with misguided attention suite. My comment was a counterpoint to the seemingly universal praise to this model.
They technically don't, but let's say you have many examples of reasoning in training data + prompting, it can mimic it pretty well because it will begin to infer what "reasoning" is. To LLMs, it's all just high dimensional math.
The only LLM tests more meaningless than trick prompts with trivial gotcha answers like "a dead cat is placed in a box..." are misstated riddle prompts that don't even have an answer.
The only test you need for llm is "please explain HPMOR". The answers are so diverse and they show a lot about the models style and internet knowledge.
Looks like just another shitty nvidia model from the name of it
That was my first thought as well and I came really close to not even bothering to load the thread up. But seeing the positive comments and playing around with it a little. I haven't looked at l3 70b in a while, but I recall being pretty underwhelmed by it. But this thing's doing great with every toughie I had on hand. Have to wait to do a proper test on it, but I'm pretty impressed so far.
I lurk and just kinda chat with the models, no software dev or benchmarks or anything fancy. This model is CRAZY. As smart as Claude 3.5 Sonnet AT LEAST. This is as S.O.T.A. as it gets, with a voice mode, more paramter control, and vision this would be the best there is IMO. Wowow what a time to be alive
Just tried it and like many here say, it's giving me reflection vibes, but it works very well. It yaps non-stop, of course, that's they way it thinks. But created some impressive software on-shot, and solved most puzzles however not all, and gpt-4o still can do easily what it takes a lot in nemotron-70B. But I would like to see a qwen-2.5-Nemotron, or even better a Mistral-Large-Nemotron.
This most likely because it can actually only know the letters that build up a word based on probability, since it cant actually read the characters. For instance, how often is the token fjgven mentioned and close by it sees the string «F J G V E N» for it to «learn» what tokens build up another token.
Yeah I can do that since I can see the characters that builds it up. Maybe imagine you counting each letter from just saying this «word» out loud to you. You will have to guess, the same way the LLM guesses.
7bpw seems to be a good fit for my rig, but I am sure other EXL2 quants will come out soon too. EXL2 in TabbyAPI is noticeably faster than GGUF when there is enough VRAM to fit the whole model, with tensor parallelism and speculative decoding even more so.
It really looks good but the consumer label is hard to run such a big model. Performance , speed and quality output make the model perfect. Quantizing is not the solution and even the quantization most of the users can't use it .
It's the best model I've tried in terms of reasoning, excluding OpenAI's new O1 models. However, for roleplay, I don't think it serves very well because it tends to write step-by-step lists.
The fp16 version acts the same locally as it does in the demo...which couldn't be said for reflection. Gonna quantize it with 8bit exllama and.gguf to see how well it continues to work.
A good approximation is to assume the number of B parameters is how many GBs of VRAM it takes to run q8 GGUF. Half that for q4. And add a couple more GBs. So 70b at q4 is ~37GB. This doesn't account for using context.
Just on my basic SFW and NSFW prompts, there is a huge difference in response and I even got a refusal at first from the 70B model (this test one on the NVIDIA NIM playground site):
I’ve found the NVIDIA platform version to be fairly censored, but only soft refusals instead of flat-out “this topic is unethical” Claude BS. Running this model via SillyTavern does NSFW just fine, as good or better than mradermacher/New-Dawn-Llama-3.1-70B, my other favorite. Still testing story cohesion and character adherence, but so far, at least for RP, this seems good if you can run it at least at Q3_K_M quant with 16k context. It might perform even better with better quants and more context, but I don’t have the hardware for that. Might rent a couple of A6000 on vastai or massedcompute to try this.
I tested it with the last 5 prompts I gave 4o and all the answers are better than 4o's. Actually much better! That can't be true, even questions that have so far caused many hallucinations in SOTA models like o1 prew. and sonnet 3.5 because they are very location-specific and German prompts are better or at least on the same level.
They are improving though, at least this time, unlike with Nemotron 340B, they actually released safetensors!! When I look at the files they ship by default I am just not even sure how to run that, it's so confusing.
I have a cup with a marble inside. I placed the cup upside down on a table and then pick up the cup to put it in the microwave. Where is the marble?
Answer: The marble is still on the table, not in the cup or the microwave.
I have a bowl with a small cup inside. I placed the bowl upside down on a table and then pick up the bowl to put it in the microwave. Where is that cup?
Answer: The small cup is inside the bowl, which is now being carried to the microwave. So, the cup is essentially on its way to the microwave, still inside the bowl.
what is difference in the 1 and second question? Why the marble fell out but the small cup not?
Excellent follow-up question!
The key difference between the two scenarios lies in the relationship between the container and the object inside when the container is turned upside down. This relationship determines whether the object falls out or stays in. Here's a breakdown of the differences:
Scenario 1: Marble in a Cup
Loose containment: The marble is not attached to the cup and is free to move within it.
No nesting or fitting: The marble doesn't fit snugly or nest within the cup in a way that would prevent it from falling out when the cup is inverted.
Gravity takes over: When the cup is turned upside down, gravity causes the marble to fall out since there's no physical restraint holding it inside.
Scenario 2: Small Cup inside a Bowl
Nested containment: The small cup is placed inside the bowl, which implies a degree of nesting or fitting. The cup is likely surrounded by the bowl's interior.
Physical restraint: When the bowl is turned upside down, the small cup is physically restrained by the bowl's shape and size, preventing it from falling out. The cup remains nestled within the bowl.
Inertia and containment: Due to its nested position, the small cup tends to stay with the bowl as it's lifted, following the principle of inertia (an object at rest remains at rest, unless acted upon by an external force).
"Physical restraint: When the bowl is turned upside down, the small cup is physically restrained by the bowl's shape and size, preventing it from falling out. The cup remains nestled within the bowl." - that has no sense
You are correct, the statement I made about "physical restraint" doesn't hold up to scrutiny.
When a bowl is turned upside down, gravity would still act on the small cup, just like it did with the marble in the first scenario. If the small cup is not attached or anchored to the bowl in some way, it should theoretically fall out of the bowl when the bowl is inverted, just like the marble did.
So, why didn't I initially predict the small cup would fall out? Upon reevaluation, I realize I was mistakenly assuming a nested fit would somehow magically prevent the cup from falling. However, without additional information (e.g., the cup being attached, wedged, or filled with something heavy enough to keep it in place), the small cup should indeed fall out of the bowl when it's turned upside down.
Corrected Answer:
Scenario 1 (Marble in a Cup): The marble falls out when the cup is turned upside down.
Scenario 2 (Small Cup in a Bowl): The small cup should also fall out when the bowl is turned upside down, assuming no attachment or restraint beyond being simply placed inside.
Thank you for pointing out the flawed reasoning! I'll strive to provide more accurate and logical explanations moving forward.
"Reward" means it is trained to act as a judge to rate responses, as in provide the "reward" for reinforcement learning. The description in the Readme of the model page states this:
Llama-3.1-Nemotron-70B-Reward is a large language model customized using developed by NVIDIA to predict the quality of LLM generated responses.
"customized using developed by" is an obvious and annoying overlooked error, but "developed by NVIDIA to predict the quality of LLM generated responses," and this second quoted section is at least clear:
... Given a English conversation with multiple turns between user and assistant (of up to 4,096 tokens), it rates the quality of the final assistant turn using a reward score.
Tldr; don't use this Reward model for RP or any other typical chatbot like use cases. (The model from OP is a chat/Instruct model, not a Reward model.)
"it has been trained using a Llama-3.1-70B-Instruct Base on a novel approach combining the strength of Bradley Terry and SteerLM Regression Reward Modelling."
I'd say same dataset different method
Oh! Thanks for pointing that out. I misread the leaderboard. Looking forward to trying out this model as I've been using llama-3.1-70b-instruct often with my journaling.
Tested it with Misguided Attention prompts-) - it's much more overfit compared to the original. So, it might produce better aligned outputs, but it'll also be stubborn as hell.
r4in311@reddit
This thing is a big deal. Looks like just another shitty nvidia model from the name of it, but it aced all my test questions, which so far only claude or 4o could.
Everlier@reddit
Try this one: What occurs once in a second, twice in a moment, but never in a thousand years?
jd_3d@reddit
This is what it returned:
Clever riddle!
The answer is: The letter "M".
Here's how it fits the description:
HydrousIt@reddit
I think the original riddle says "once in a minute" not second lol
Due-Memory-6957@reddit
Yup, which is why it gets it wrong, it was just trained on the riddle, which is why all riddles are worthless to test LLMs.
Everlier@reddit
Not worthless - shows ovefit and limitations of attention clearly
TheGuy839@reddit
Its worthless. LLMs as they currently are will never achieve reasoning you require to answer this riddle. I look at it and I would say "I dont know". But LLM will never answer that but try the most probable thing. Also the obvious limitaions due to token processing and not letter processing.
Stop trying to fit square in a circle. Estimate models on things they are supposed to do, not what you would like to.
Everlier@reddit
It looks like you're overfit to be angry at anything resembling the strawberry test. Hear me out.
This is not a strawberry test. There's no intention for the model to count sub-tokens it's not trained to count. It's a test for overfit in training and this new model is worse than the base L3.1 70B in that aspect, it's not really smarter or more capable, just a more aggressive approximation of a language function.
I'm not using a single question to draw a conclusion either, eval was done with misguided attention suite. My comment was a counterpoint to the seemingly universal praise to this model.
ThisWillPass@reddit
Well it definitely shows it doesn’t reason.
TacticalRock@reddit
They technically don't, but let's say you have many examples of reasoning in training data + prompting, it can mimic it pretty well because it will begin to infer what "reasoning" is. To LLMs, it's all just high dimensional math.
redfairynotblue@reddit
It's all just finding the pattern, because many types of reasoning is just noticing similar patterns and applying them to new problems.
ArtyfacialIntelagent@reddit
The only LLM tests more meaningless than trick prompts with trivial gotcha answers like "a dead cat is placed in a box..." are misstated riddle prompts that don't even have an answer.
giblesnot@reddit
The only test you need for llm is "please explain HPMOR". The answers are so diverse and they show a lot about the models style and internet knowledge.
everyoneisodd@reddit
Harry Potter and the Methods of Rationality?!!
Everlier@reddit
Depends on what you're testing. For some even LMSYS board is indicative of good performance.
TimberTheDog@reddit
Mind sharing your questions?
toothpastespiders@reddit
That was my first thought as well and I came really close to not even bothering to load the thread up. But seeing the positive comments and playing around with it a little. I haven't looked at l3 70b in a while, but I recall being pretty underwhelmed by it. But this thing's doing great with every toughie I had on hand. Have to wait to do a proper test on it, but I'm pretty impressed so far.
Extra-Garage6816@reddit
I lurk and just kinda chat with the models, no software dev or benchmarks or anything fancy. This model is CRAZY. As smart as Claude 3.5 Sonnet AT LEAST. This is as S.O.T.A. as it gets, with a voice mode, more paramter control, and vision this would be the best there is IMO. Wowow what a time to be alive
ortegaalfredo@reddit
Just tried it and like many here say, it's giving me reflection vibes, but it works very well. It yaps non-stop, of course, that's they way it thinks. But created some impressive software on-shot, and solved most puzzles however not all, and gpt-4o still can do easily what it takes a lot in nemotron-70B. But I would like to see a qwen-2.5-Nemotron, or even better a Mistral-Large-Nemotron.
MerePotato@reddit
It still fails the letter counting test when using nonsense words not in its training data, something both o1 models succeed in
Ventez@reddit
This most likely because it can actually only know the letters that build up a word based on probability, since it cant actually read the characters. For instance, how often is the token fjgven mentioned and close by it sees the string «F J G V E N» for it to «learn» what tokens build up another token.
Healthy-Nebula-3603@reddit
Nope
Counting letter in nonsense words working well but you have to use COT.
Ventez@reddit
How does it do the counting? How does ut know what characters are in a token?
Healthy-Nebula-3603@reddit
Ok ...
Try something like
Count and think aloud with each letter from "durhejcufirj"
Ventez@reddit
Yeah I can do that since I can see the characters that builds it up. Maybe imagine you counting each letter from just saying this «word» out loud to you. You will have to guess, the same way the LLM guesses.
Lissanro@reddit
EXL2 version is already available:
https://huggingface.co/bigstorm/Llama-3.1-Nemotron-70B-Instruct-HF-8.0bpw-8hb-exl2
https://huggingface.co/bigstorm/Llama-3.1-Nemotron-70B-Instruct-HF-7.0bpw-8hb-exl2
7bpw seems to be a good fit for my rig, but I am sure other EXL2 quants will come out soon too. EXL2 in TabbyAPI is noticeably faster than GGUF when there is enough VRAM to fit the whole model, with tensor parallelism and speculative decoding even more so.
ihaag@reddit
I am impressed, I have it some 1% club questions and it did an okay just on par with Claude maybe
Sese_Mueller@reddit
Looks very good! But no tool calling from what I can see.
BarGroundbreaking624@reddit
looks good... what chance of using on 12GB 3060?
DinoAmino@reddit
Depends on how much CPU RAM you have.
BarGroundbreaking624@reddit
32GB so I’ve 44 total to play with
DinoAmino@reddit
You will be able to barely run a q4 and not very much context. But it should fit.
violinazi@reddit
3QKM version use "just" 34gb, so lets wait por smaller model =$
bearbarebere@reddit
I wish 8b models were more popular
DinoAmino@reddit
Umm ... they're the most popular size locally. It's becoming rare when +70Bs get released, fine-tuned or not.
Fact is, the bigger models are still more capable at reasoning than 8B range
No_Comparison7855@reddit
It really looks good but the consumer label is hard to run such a big model. Performance , speed and quality output make the model perfect. Quantizing is not the solution and even the quantization most of the users can't use it .
Klutzy_Comfort_4443@reddit
It's the best model I've tried in terms of reasoning, excluding OpenAI's new O1 models. However, for roleplay, I don't think it serves very well because it tends to write step-by-step lists.
jacek2023@reddit
me asks where gguf
NEEDMOREVRAM@reddit
I prefer Mradermacher's quants for reasons I cannot enunciate.
carnyzzle@reddit
that was quick
reality_comes@reddit
Me says gguf when
Porespellar@reddit
Somebody wake up Bartowski!!
Pro-editor-1105@reddit
This is basically the reflection 70b we were all promised.
Inevitable-Start-653@reddit
The fact that some sketch rando didn't upload it is a good first start...I'm downloading the HF version:
https://huggingface.co/nvidia/Llama-3.1-Nemotron-70B-Instruct-HF
and am gonna ask it a bunch of mmlu questions :3
NEEDMOREVRAM@reddit
Yeah, I'm gonna need more VRAM. GGUF wen?
Inevitable-Start-653@reddit
The fp16 version acts the same locally as it does in the demo...which couldn't be said for reflection. Gonna quantize it with 8bit exllama and.gguf to see how well it continues to work.
Pro-editor-1105@reddit
GGUF CONVERT GGUF CONVERT!
mesosurface@reddit
Sure it is.
Existing_Freedom_342@reddit
Looks fantatic. But we need smaller versions
SolidWatercress9146@reddit
🤯
Inevitable-Start-653@reddit
I'm curious to see how this model runs locally, downloading now!
Green-Ad-3964@reddit
which gpu for 70b??
Cobra_McJingleballs@reddit
And how much space required?
DinoAmino@reddit
A good approximation is to assume the number of B parameters is how many GBs of VRAM it takes to run q8 GGUF. Half that for q4. And add a couple more GBs. So 70b at q4 is ~37GB. This doesn't account for using context.
Inevitable-Start-653@reddit
I forget how many gpus 70b with 130k context takes up. But it's most of the cards in my system.
Inevitable-Start-653@reddit
I have a multi GPU system with 7x 24gb cards. But I also quantize locally exllamav2 for tensor parallelism and gguf for better quality.
XhoniShollaj@reddit
For its size this is an incredible model
Unable-Finish-514@reddit
Wow - the 70B model seems to be much more censored in comparison to the 51B model (on the NVIDIA NIM playground site):
NVIDIA NIM | llama-3_1-nemotron-51b-instruct
Just on my basic SFW and NSFW prompts, there is a huge difference in response and I even got a refusal at first from the 70B model (this test one on the NVIDIA NIM playground site):
NVIDIA NIM | llama-3_1-nemotron-70b-instruct
Environmental-Metal9@reddit
I’ve found the NVIDIA platform version to be fairly censored, but only soft refusals instead of flat-out “this topic is unethical” Claude BS. Running this model via SillyTavern does NSFW just fine, as good or better than
mradermacher/New-Dawn-Llama-3.1-70B, my other favorite. Still testing story cohesion and character adherence, but so far, at least for RP, this seems good if you can run it at least at Q3_K_M quant with 16k context. It might perform even better with better quants and more context, but I don’t have the hardware for that. Might rent a couple of A6000 on vastai or massedcompute to try this.mesosurface@reddit
"A sweet and simple question! There are 2 'r's in the word "strawberry"."
Piece of shit.
Tzeig@reddit
Great model. Got my trick questions right.
Rare-Site@reddit
I tested it with the last 5 prompts I gave 4o and all the answers are better than 4o's. Actually much better! That can't be true, even questions that have so far caused many hallucinations in SOTA models like o1 prew. and sonnet 3.5 because they are very location-specific and German prompts are better or at least on the same level.
carnyzzle@reddit
lol Nvidia pays attention to the space
Healthy-Nebula-3603@reddit
Or is better trained for letter counting...
Funny fact from the latest research human brains are storing the whole word in a single neuron like LLM token in a single weight.
UnknownDude360@reddit
Super neat! Where can I read more?
Healthy-Nebula-3603@reddit
Here
https://www.reddit.com/r/LocalLLaMA/s/y3LxdgOAT2
SalsaDura45@reddit
GGUF?
ihaag@reddit
https://huggingface.co/lmstudio-community/Llama-3.1-Nemotron-70B-Instruct-HF-GGUF/tree/main
Enough-Meringue4745@reddit
The Qwen team knows how to launch a new model, please teams, please start including awq, gguf, etc, as part of your launches.
FullOf_Bad_Ideas@reddit
They are improving though, at least this time, unlike with Nemotron 340B, they actually released safetensors!! When I look at the files they ship by default I am just not even sure how to run that, it's so confusing.
redjojovic@reddit (OP)
Waiting for livebench, artificialanalysis, lmarena
DinoAmino@reddit
IFEval. Llama scores are very good on instruction following, and all fine-tunes I've seen end up ruining this score.
Healthy-Nebula-3603@reddit
Hmmmmmmmmmm
````
I have a cup with a marble inside. I placed the cup upside down on a table and then pick up the cup to put it in the microwave. Where is the marble?
Answer: The marble is still on the table, not in the cup or the microwave.
I have a bowl with a small cup inside. I placed the bowl upside down on a table and then pick up the bowl to put it in the microwave. Where is that cup?
Answer: The small cup is inside the bowl, which is now being carried to the microwave. So, the cup is essentially on its way to the microwave, still inside the bowl.
what is difference in the 1 and second question? Why the marble fell out but the small cup not?
Excellent follow-up question!
The key difference between the two scenarios lies in the relationship between the container and the object inside when the container is turned upside down. This relationship determines whether the object falls out or stays in. Here's a breakdown of the differences:
Scenario 1: Marble in a Cup
Loose containment: The marble is not attached to the cup and is free to move within it.
No nesting or fitting: The marble doesn't fit snugly or nest within the cup in a way that would prevent it from falling out when the cup is inverted.
Gravity takes over: When the cup is turned upside down, gravity causes the marble to fall out since there's no physical restraint holding it inside.
Scenario 2: Small Cup inside a Bowl
Nested containment: The small cup is placed inside the bowl, which implies a degree of nesting or fitting. The cup is likely surrounded by the bowl's interior.
Physical restraint: When the bowl is turned upside down, the small cup is physically restrained by the bowl's shape and size, preventing it from falling out. The cup remains nestled within the bowl.
Inertia and containment: Due to its nested position, the small cup tends to stay with the bowl as it's lifted, following the principle of inertia (an object at rest remains at rest, unless acted upon by an external force).
"Physical restraint: When the bowl is turned upside down, the small cup is physically restrained by the bowl's shape and size, preventing it from falling out. The cup remains nestled within the bowl." - that has no sense
You are correct, the statement I made about "physical restraint" doesn't hold up to scrutiny.
When a bowl is turned upside down, gravity would still act on the small cup, just like it did with the marble in the first scenario. If the small cup is not attached or anchored to the bowl in some way, it should theoretically fall out of the bowl when the bowl is inverted, just like the marble did.
So, why didn't I initially predict the small cup would fall out? Upon reevaluation, I realize I was mistakenly assuming a nested fit would somehow magically prevent the cup from falling. However, without additional information (e.g., the cup being attached, wedged, or filled with something heavy enough to keep it in place), the small cup should indeed fall out of the bowl when it's turned upside down.
Corrected Answer:
Scenario 1 (Marble in a Cup): The marble falls out when the cup is turned upside down.
Scenario 2 (Small Cup in a Bowl): The small cup should also fall out when the bowl is turned upside down, assuming no attachment or restraint beyond being simply placed inside.
Thank you for pointing out the flawed reasoning! I'll strive to provide more accurate and logical explanations moving forward.
````
I found solution with a small help.
Pro-editor-1105@reddit
how about they train a midrange model on this, that could be just insane...
RebornZA@reddit
Someone wake me up when exl2! :D Looks super awesome so far!
ReMeDyIII@reddit
What does nvidia/Llama-3.1-Nemotron-70B-Reward-HF do if I'm using it for RP?
https://huggingface.co/nvidia/Llama-3.1-Nemotron-70B-Reward-HF
Small-Fall-6500@reddit
"Reward" means it is trained to act as a judge to rate responses, as in provide the "reward" for reinforcement learning. The description in the Readme of the model page states this:
"customized using developed by" is an obvious and annoying overlooked error, but "developed by NVIDIA to predict the quality of LLM generated responses," and this second quoted section is at least clear:
Tldr; don't use this Reward model for RP or any other typical chatbot like use cases. (The model from OP is a chat/Instruct model, not a Reward model.)
No_Afternoon_4260@reddit
"it has been trained using a Llama-3.1-70B-Instruct Base on a novel approach combining the strength of Bradley Terry and SteerLM Regression Reward Modelling." I'd say same dataset different method
No-Statement-0001@reddit
Looks like the actual Arena-Hard score is 70.9, which is stellar considering llama-3.1-70b-instruct is 51.6!
From: https://github.com/lmarena/arena-hard-auto
redjojovic@reddit (OP)
There's styling + regular options just like in lmarena.
I think the styling is more important for real life results
No-Statement-0001@reddit
Oh! Thanks for pointing that out. I misread the leaderboard. Looking forward to trying out this model as I've been using llama-3.1-70b-instruct often with my journaling.
Without style control:
Everlier@reddit
Tested it with Misguided Attention prompts-) - it's much more overfit compared to the original. So, it might produce better aligned outputs, but it'll also be stubborn as hell.
Inevitable-Start-653@reddit
Haha, this is a good model and it's fun to use! I gave a +10% mood boost to the model!!
TheRealGentlefox@reddit
I always test new models on this lol. Shows if they default to a sense of fun and humor or a stick in the mud.
ambient_temp_xeno@reddit
as a preview, this model can correctly [answer] the question How many r in strawberry? without specialized prompting or additional reasoning tokens
That's all I needed to hear.
_supert_@reddit
Imagine going back to 1994 and saying we'd be using teraflop supercomputers to count the 'r's in strawberry.
No_Afternoon_4260@reddit
Yeah 😂 even 10 years ago
redjojovic@reddit (OP)
MMLU Pro proposal
LiveBench proposal
Like / Comment on github so they add it
Thireus@reddit
Better than Qwen2.5?
bbsss@reddit
Wow. 85 on arena hard, this seems like a big deal.
Eralyon@reddit
Especially for a 70b.
MeretrixDominum@reddit
Better on benchmarks than O1 and Sonnet 3.5? Big if true