Haters gonna hate.
"All that being said: if applying reflection fine-tuning drives a similar jump in eval performance on Llama 3.1 405B, we expect Reflection 405B to achieve near SOTA results across the board."
I don't hate the guy though. But when someone makes us all look like fools it is not appreciated. And he still hasn't provided us with anything. You're acting like this tweet somehow solves the fact that he still hasn't provided anything to us.
Look at what it actually says:
> These were tested on a private API version and not an open-weights version.
So how do we know this isn't a model with a custom prompt?
Why not just open the directory and make a torrent? It would solve everything in like 5 minutes.
Not really. The "how bad this was" are still easily winning considering we have seen 0 open weights and are provided some ambiguous results from an API that we have no clue the validity of.
Open weights or GTFO.
It has been fucking 6 hours since he trained the model, give the man a fucking break, and guess what he released the weights? Normally I don't get this angry but holy shit you people are fucking insane.
I just went off on a guy for calling someone Matt.
I'm not Matt, but doxxing isn't funny.
Like you might be right that this is all just bullshit/scam.
But you are attacking people for reserving their judgement. That is disgusting mob mentality.
Fair concerns. For all we know, under the hood this API could redirect queries to Claude-3.5 Sonnet with a specific system prompt, or another SotA proprietary model.
I don't understand why people need to pile on the hate so quick. Is it really that hard to just reserve judgment for a few weeks and see what comes of it? This avenue of applying more test time compute is a very promising direction to me and could be a great way for open source models to exceed closed models that don't want to spend the $$$ on each request.
We all tried it, it's performance on real world tasks is terrible despite the high benchmarks. Maybe the model is still broken in some way like they've been claiming and really is good but I don't see it.
I disagree here.
Gemma was a novel architecture with an attention mechanism that wasn't well supported.
This is a fine-tune of Llama. There is nothing to resolve, they're playing us for fools.
Bro, seriously? Man, aside from the supposedly broken model and the super-duper-secret private api shit: this guy didn't know the difference between llama 3 and llama 3.1
I mean, by now even my grandmother should know the difference.
The model he posted on huggingface was absolutely not broken, it was simply a llama-3, as you would expect from a llama-3. There is zero evidence that there was anything wrong with him.
I'm slowly coming to the conclusion that this guy is either stupid and narcissistic enough to believe he can fool the world in such a simple way - or, another possible explanation could be: he himself has been the victim of a scam. Perhaps he doesn't have direct access to the backend of this ominous private API himself. Perhaps he still hasn't realized that he has been misused as a puppet and ruined economically and in terms of marketing with this action. It wouldn't be the first time that someone had economic enemies and fell into a trap.
The whole thing is highly suspicious and whether he is a victim himself or not, whether he is stupid or not: he clearly also seems to want to lie and hide things! So there are neither excuses nor pity for this egomaniacal behavior.
Holy shit you are arrogant. Maybe he made a typo maybe his computer malfunctioned idno just stfu and wait for a response before spouting bullshit? Talk about narcissism reread your comment and reflect.
I made some quick tests with this model yesterday and actually it performed not that bad. But I can't compare it with Claude or OpenAI, I don't use them.
[https://huggingface.co/bartowski/Reflection-Llama-3.1-70B-GGUF/blob/main/Reflection-Llama-3.1-70B-Q4\_K\_M.gguf](https://huggingface.co/bartowski/Reflection-Llama-3.1-70B-GGUF/blob/main/Reflection-Llama-3.1-70B-Q4_K_M.gguf)
Don't you guys get that the model is broken, he said? The tests were based on private API, now yeah, you might not trust that the model, maybe model behind this is different, then it's okay, but as per Matt, the model everyone downloaded from HF is broken, and yeah, I tried too, it is far worse than LLaMA 3.1 70b.
I mean it's a diabolical plan: Release an "open" model that crushes benchmarks, but then don't actually release working weights and instead just point to your "private API" that produces those results.
I can't test his "private API" can I? The whole thing smells bad, as far as I'm concerned this is a publicity stunt to advertise his LLM service.
I'm an open source leaderboard maintainer, without weights any test results are just a free ad for his service.
I do benchmark the big APIs for reference but no interest in starting to do it for every tom dick and harry, when weights are fixed I'll try again.
No need to guess, it is now publicly accessible on hf;
[Reflection 70B llama.cpp (Correct Weights) - a Hugging Face Space by gokaygokay](https://huggingface.co/spaces/gokaygokay/Reflection-70B-llamacpp)
Perhaps somebody with an X account could request a prompt inquiring the model about its identity at [this X post from a user with \~180,000 X followers ](https://x.com/DotCSV/status/1832702433329389839)who purportedly [has been](https://x.com/mattshumer_/status/1832566622801780865) given API access to the good model by Matt Shumer.That account has posted a number of purported responses to various prompts by the good model.
What would the point be?
I get that they want to declare they have a great model based on using their platform to generate data, and everyone is just saying it's a scam or trick, but think it through. No one will just believe it until others third parties have independently verified it, which several will. And if everyone disproves it, then it will massively harm the valuation and growth of the company they are trying to promote.
I'm not saying I automatically think the model is amazing, although the concept is built on strong donations and has been around for a while, I'm just saying it would be a really bad publicity stunt and a huge reputational risk.
I fully understand why someone would like and scam... But lying about something to everyone at once in a community that tests and communicates within hours of a release, about something where the claims can be disproven and widely reported... Seems like a scam that does nothing apart from having a negative effect on reputation.
Cool... I should have mentioned my latest fine tune gets 101% on all benchmarks, and also created its own benchmark... If you want me to tell you the HF model name just send me a bitcoin
With how scam works in a few days he will say he almost got it working but there are still issues and he needs addition two weeks and so on and so on. It works for that Italian guy who peddles cold fusion for decades now.
He quietly released this https://huggingface.co/mattshumer/ref_70_e3, he said he was going to run the benchmarks again before re-announcing it.
If he's aiming to drag things out he's doing a pretty shit job.
I was thinking this earlier! It would be a clever con. I was thinking maybe it’s using the OpenAI fine tuning service. Until we get weights that equal what they have in their benchmarks I guess it’s a possibility.
Really? To me, it felt way too dumb to be Claude. It pretty much was llama 3.1 70b in behavior - I struggled to find any obvious real world question performance above it.
This conversations reminds me that [somebody](https://www.reddit.com/r/LocalLLaMA/comments/1f9um6s/comment/llpdj5x/) noticed that the demo made calls to an endpoint called "openai\_proxy" while I was one of the people explaining why that might not be as suspicious as it sound on the surface, I'm now starting to seriously think it might actually have been exactly what it sounded like.
The fact that he has decided to retrain the model instead of just uploading the working model he is hosting privately is just not logical at all unless he literally cannot upload the private model. Which would be the case if he is just proxying another model.
He claims there isn't any Anthropic data.
https://x.com/mattshumer_/status/1832203011059257756#m
( if I had more time on the playground, I could've confirmed whether it was Claude or not :\ )
This reminds me that [somebody](https://www.reddit.com/r/LocalLLaMA/comments/1f9um6s/comment/llpdj5x/) noticed that the demo made calls to an endpoint called "openai\_proxy" while I was one of the people explaining why that might not be as suspicious as it sound on the surface, I'm now starting to seriously think it might actually have been exactly what it sounded like.
The fact that he has decided to retrain the model instead of just uploading the working model he is hosting privately is just not logical at all unless he literally cannot upload the private model. Which would be the case if he is just proxying another model.
Different prompt, temperature, etc. The simple fact is that they haven't released the "good" version of their model and have no reason to. This should be a 30 minute fix on the HuggingFace repo, no reason for it to not be available already.
Also this isn't a full replications of their results, on the original post they claimed it beat other models on almost everything and we see it isn't quite like that.
Until the open weights perform just as well as this suspiciously private, researcher only API we are better off staying skeptical. Still looks like a scam to me.
It almost replicated except MMLU (2% behind), "MMLU: 87% (in line with Llama 405B), GPQA: 54%, Math: 73%." Quite close to Sonnet and other SOTA. But it is okay, there is something he's definitely hiding, but I kinda feel this is really achieved by them with reflection. Let's wait and see.
The model he posted to hugging face has been shown to be llama-3 with a Lora applied, he's claiming it's definitely llama-3.1 and he doesn't know what a Lora is.
The "private API" could be any model and I see no reason to trust these apparently 3rd party benchmarkers.
This is 100% a scam
Guy with 0 background, no idea what LORA is, "wrong" weights uploaded, "wrong" model name promoted, "my cat ate my model i'll release the real one next week", does not disclose he has ownership in the company he promotes, the model outputs garbage with 4x more tokens generation, sounds legit to me. :)
He knows what it is check his post history. He didn’t understand “LORAing” in the context used. He stated his ownership in the company is a $1000 investment lol.
Yeah soo likely that it’s a wrapper and two guys with reputation and multiple companies are going to lie about it and ruin their lives for literally zero reason.
Surely you can see that it is way more likely they used a dataset generated from claude to create the reflection template.
And thinks rules don't apply to big shot entrepreneurs like him.
"I Couldn't Play By The Rules, So I Became an Entrepreneur", by Matt Schumer.
https://www.newsweek.com/i-couldnt-play-rules-so-i-became-entrepreneur-opinion-1448286
He knows what a lora is and you can check his history to see him using them. He was talking specifically about the term “LORAing” in the context. 0% chance its a scam it wouldn’t make sense to risk his reputation on something easily disproven
https://www.reddit.com/r/LocalLLaMA/comments/1fcerck/reflection_ref_70_e3_refuses_to_output_meta_tag/
I've been doing some testing, but the community does not seem interested in objective facts. My post keeps getting downvoted, I'm sure it will be off the front page soon.
Yeah that's the point, there was the initial announcement of it, then some researchers were like "We are somewhat able to replicate the results", but then it was eventually proven to not work
Two years ago(?) there was a paper / video of a supposed room temperature superconductor (they had a sweet floating rock too). And everyone was like "Yeah that's bullshit." But then some hobby chemists were like "Actually I managed to recreate a small port of it from their paper, and it floats too." and this started a race to recreate it by a lot of laboratories around the world. At the end it was not a room temperature superconductor but they managed to find some new stuff.
> does not suffer from the issues with the version publicly released on Hugging Face
It's not rocket science to upload a model to Hugging Face. It's very suss that they can't seem to upload a BF16 or GGUF of a fine tuned Llama to Hugging Face that can be properly tested.
> The model seems to be achieving these results through forcing an output ‘reflection’ response where the model always generates scaffolding of <thinking>, <reflection>, and <output>. In doing this it generates more tokens than other models do on our eval suite with our standard ‘think step by step’ prompting.
> For example, it appears that Reflection 70B is not capable of ‘just responding with the answer’ in response to an instruction to classify something and only respond with a one word category.
One can always add a postprocesssor on top of Reflection to filter out everything before \<output\>, problem solved. I don't like this nitpicking. Who cares if a model outputs 10 tokens or 100? Is the answer correct or not??
For real time inference is the real issues, your time to first token jumps by a huge margin if you have to wait for 2000 tokens to be generated of the model reflecting. Might also explain why the cloud providers haven’t adopted it yet.
[https://huggingface.co/mattshumer/Reflection-Llama-3.1-70B-ep2-working](https://huggingface.co/mattshumer/Reflection-Llama-3.1-70B-ep2-working) It seems an Epoch 2 model was released a few hours ago silently: "Epoch 2, still finishing up Epoch 3. This should be slightly less powerful, but still pretty close." [https://x.com/DavidFSWD/status/1832826469590130955](https://x.com/DavidFSWD/status/1832826469590130955) someone confirmed it is working perfectly
nope bro, stop hating now, it is released now, open router: [https://x.com/mattshumer\_/status/1832881734230192442](https://x.com/mattshumer_/status/1832881734230192442) , [https://huggingface.co/mattshumer/ref\_70\_e3](https://huggingface.co/mattshumer/ref_70_e3) hugging face
Downloading this now:
https://huggingface.co/mattshumer/Reflection-Llama-3.1-70B-ep2-working
can run locally with my own setup, and am interested in testing it out!
Thanks!! Will download this one now ☺️ for all the downloading this still isn't anywhere near as bad as llama405b...that sucker was a multi day download and I needed to download it twice after they updated their repo too.
I wouldn't bother, it doesn't even try to output the tags the "broken" model outputs, so no idea what they mean by "working".
https://preview.redd.it/wgvki1etpmnd1.png?width=1792&format=png&auto=webp&s=85ac2c1c112810a61f861807a9f3ee0310b10cc5
Hmm 🤔 ...this whole saga is so strange. Download will finish in a lil bit, I've got to try it out, I got the very first upload to work but only could get one response out.
You would not push false hype again?
Why are ppl upvoting this again.
Are "The boy who cry wolf" that obscure of a story, or do you also have bot to upvote yourself?
https://www.reddit.com/r/LocalLLaMA/comments/1fa4y7q/first_independent_benchmark_prollm_stackunseen_of/?utm_source=share&utm_medium=mweb3x&utm_name=mweb3xcss&utm_term=1&utm_content=share_button
Given the impressive translation capabilities of LLMs, wouldn't it be possible to utilize a new, highly compressed language for the reasoning process and then switch to natural language for the final answer?
It's brilliant at zero shot reasoning tasks , would definitely try out more difficult mcq questions .
I don't know how but the model I used via open router was fantastic and could not relate to its criticism.
It gave me better responses than other top models , I only tried out zero shot reasoning tasks though
No story telling,coding etc
I thought the difference was from the different prompt method. Why didn't they just use the released version with the refection's default system prompt like they used now?
The whole thing is very weird and annoying. They supposedly uploaded the model to HF incorrectly, so naturally the solution was to completely redo the finetune? I have no idea.
There are public weights, but they are subpar. And this test was not performed on them:
>Since our first testing of the public release version of Reflection Llama 3.1 70B, [u/mattshumer\_](https://x.com/mattshumer_) has shared access to a privately hosted version of the model that does not suffer from the issues with the version publicly released on Hugging Face.
Now why Matt can't just upload the privately hosted version to HF is a mystery. But he has instead opted to retrain the model and is currently posting new checkpoints to HF.
[https://huggingface.co/mattshumer/Reflection-Llama-3.1-70B-ep2-working](https://huggingface.co/mattshumer/Reflection-Llama-3.1-70B-ep2-working) It seems an Epoch 2 model was released a few hours ago silently: "Epoch 2, still finishing up Epoch 3. This should be slightly less powerful, but still pretty close." [https://x.com/DavidFSWD/status/1832826469590130955](https://x.com/DavidFSWD/status/1832826469590130955) someone confirmed it is working perfectly
I only care about this for the possibility too generate better synthetic data with step by step reasoning
Other than that there is no point in making the token consumption exponential
"Oh, the benchmark didn't work? Let's see what tests you used..."
*Scrambles to train the model on the test data*
"Here you go, try the private API version"
"When using Reflection’s default system prompt and extracting answers only from within Reflection’s <output> tags, results show substantial improvement: MMLU: 87% (in-line with Llama 405B), GPQA: 54%, Math: 73%."
Actually, what is presented on the chart is based on their standard system prompt. It achieves performance close to Claude 3.5's sonnet-like output with the Reflect system prompt. If Groq hosts it, latency will not be an issue. We're just waiting for the actual weights to be released
Will not match with it for sure. I tried many different system prompts, verbose thinking output + "step by step" at the prompt, but it couldn't pass any of my expert-level coding tests from Edabit, even the 405B failed one; GPT4 too. But the model (when the demo was live) in their demo nailed all of them.
I only got one or two replies off the demo before it got "overloaded" and turned off. It seemed *alright*. The demo on hyperbolic was absolute garbage and the model forgot about its COT tags within a few messages.
All in all.. it seems like this dude has been stringing everyone else along whether there is some model or not. Even if you had slow internet, the excuses and the "retraining" now doesn't make sense. Everything is maximum hype and delay.
This is the reason I am so positive about it, and defending lol, it hurts me when people say it's far worse due to a broken HF model. But yeah, we don't know for sure if the model behind the API is actually reflection 70b or not
I love how people downvoted you without any explanation as for why he hasn't just uploaded the weights. HF messed it up? Sure buddy, then upload it to literally anywhere else or make a torrent.
The chart below is based on our standard methodology and system prompt. When using Reflection’s default system prompt and extracting answers only from within Reflection’s <output> tags, results show substantial improvement: MMLU: 87% (in-line with Llama 405B), GPQA: 54%, Math: 73%.
The chart below is based on our standard methodology and system prompt. When using Reflection’s default system prompt and extracting answers only from within Reflection’s <output> tags, results show substantial improvement: MMLU: 87% (in-line with Llama 405B), GPQA: 54%, Math: 73%.
Shumer wasn't hesitating to claim it's the "world’s top \*open-source\* model" in the initial tweet. And now some "internal" model emerges?
You certainly didn't deserve the downvotes. But the entire release event, from the beginning up to this date, was one big clusterf-k
Oh yeah super shady so far, this is a step in the right direction, hopefully he will release the corrected model in a day or two and people could start exploring what further gains there could be had with more test time compute or we know it was a scam and move on.
There is nothing toxic about questioning the validity given the inability of anyone to replicate with the released weights.
The sheer number of problems including the lack of disclosure that he is invested in both companies that he’s been saying “helped”
> These were tested on a private API version and not an open-weights version.
Admit what exactly? So far we're still correct that he hasn't given us any evidence of this being a model that he trained himself nor has he provided it to us.
All we know is that there is a closed model somewhere that is seeing some improvements that aren't verifiable by the public. How do we know it's not 405B with a custom prompt?
If you take 1 look at his twitter page for the last 2 days you can see the guy is making excuse after excuse.
> "We uploaded it but suddenly it's not the same anymore"
Give me a break, the guy could have just given us a torrent like right when drama this started but still hasn't. Why not? He even responded to someone suggesting this asking his coworker what he thought about it. Like why not just do it?
It would literally take him like 5 minutes of creating the torrent (if even) and then just seed it.
I would like to see a comparison by giving all the models similar inference compute boosts. One way to easily do this is to maybe give all the models a roughly similar token budget ( you can make multiple generations and vote for models that aren't as verbose as reflection)
144 Comments
Environmental-Car267@reddit
Many_SuchCases@reddit
alongated@reddit
RandoRedditGui@reddit
alongated@reddit
showdontkvell@reddit
alongated@reddit
showdontkvell@reddit
StartledWatermelon@reddit
dalkef@reddit
Hatter_The_Mad@reddit
jd_3d@reddit (OP)
kryptkpr@reddit
alongated@reddit
kryptkpr@reddit
alongated@reddit
Evening_Ad6637@reddit
alongated@reddit
muxxington@reddit
Sadman782@reddit
nero10578@reddit
kryptkpr@reddit
Environmental-Car267@reddit
kryptkpr@reddit
alongated@reddit
ilangge@reddit
reevnez@reddit
Wiskkey@reddit
dotcsv@reddit
Sm0g3R@reddit
StevenSamAI@reddit
waxroy-finerayfool@reddit
StevenSamAI@reddit
cuyler72@reddit
StevenSamAI@reddit
htrowslledot@reddit
htrowslledot@reddit
Thomas-Lore@reddit
extopico@reddit
htrowslledot@reddit
ivykoko1@reddit
htrowslledot@reddit
Inevitable-Start-653@reddit
ozzeruk82@reddit
Waste-Button-5103@reddit
TGSCrust@reddit
meister2983@reddit
TGSCrust@reddit
mikael110@reddit
PraxisOG@reddit
TGSCrust@reddit
mikael110@reddit
Sadman782@reddit
h666777@reddit
Sadman782@reddit
Significant-Nose-353@reddit
cuyler72@reddit
Educational_Rent1059@reddit
Waste-Button-5103@reddit
Educational_Rent1059@reddit
Waste-Button-5103@reddit
Evening_Ad6637@reddit
gibs@reddit
ArtyfacialIntelagent@reddit
Inevitable-Start-653@reddit
cuyler72@reddit
Inevitable-Start-653@reddit
mckirkus@reddit
cuyler72@reddit
physalisx@reddit
Waste-Button-5103@reddit
Inevitable-Start-653@reddit
jd_3d@reddit (OP)
Inevitable-Start-653@reddit
4hometnumberonefan@reddit
hleszek@reddit
TouristDelicious8351@reddit
KillerX629@reddit
JamesAQuintero@reddit
OXKSA1@reddit
Cantflyneedhelp@reddit
RandoRedditGui@reddit
Ivo_ChainNET@reddit
synn89@reddit
celsowm@reddit
Wiskkey@reddit
celsowm@reddit
Wiskkey@reddit
ambient_temp_xeno@reddit
strubenuff1202@reddit
gamedevgrunt@reddit
ihaag@reddit
AnomalyNexus@reddit
ispeakdatruf@reddit
athirdpath@reddit
nihalani@reddit
Sadman782@reddit
physalisx@reddit
Sadman782@reddit
Inevitable-Start-653@reddit
jd_3d@reddit (OP)
Inevitable-Start-653@reddit
Sadman782@reddit
ivykoko1@reddit
Deathmax@reddit
Inevitable-Start-653@reddit
mantafloppy@reddit
Jordangnr@reddit
Thistleknot@reddit
ivykoko1@reddit
khubebk@reddit
ambient_temp_xeno@reddit
julioques@reddit
ambient_temp_xeno@reddit
julioques@reddit
ambient_temp_xeno@reddit
julioques@reddit
ambient_temp_xeno@reddit
mikael110@reddit
Sadman782@reddit
alongated@reddit
1889023okdoesitwork@reddit
Kathane37@reddit
This_Organization382@reddit
Sadman782@reddit
a_beautiful_rhind@reddit
Sadman782@reddit
a_beautiful_rhind@reddit
Sadman782@reddit
ILikeCutePuppies@reddit
vert1s@reddit
Many_SuchCases@reddit
vert1s@reddit
ArtyfacialIntelagent@reddit
redjojovic@reddit
redjojovic@reddit
Significant-Nose-353@reddit
htrowslledot@reddit
StartledWatermelon@reddit
htrowslledot@reddit
vert1s@reddit
Significant-Nose-353@reddit
Many_SuchCases@reddit
LiquidGunay@reddit