Thread of the "Reflection 70B" debacle by it's creator

[-]

rambat1994@reddit (OP)

Many of you know me personally, so you won’t be surprised to hear that I’m already working on a new version of Reflection. This time, I’m doing everything myself, from data generation and model training to benchmarking (this reproduction experience forced me to learn how to run major benchmarks :)) and evaluation.I’m not going to rush it, and I’m going to be incredibly thorough in my testing and validation.

If and when I get it working well, I'll share more with the community.

I’m not giving up on the idea of Reflection-Tuning. Neither should you.

Thank you all for your continued support.

it ain't over, but hopefully next round we get more robust testing and benchmarking.

Still, the whole details on the "wrong file upload" and the Claude situation is not discussed and it puts a lot of the blame on Glaive CEO. IDK

[-]

rambat1994@reddit (OP)

Why am i getting downvoted for quoting the end of the twitter thread?

I AM NOT THE CREATOR OF REFLECTION!!!

[-]

i don't think fine tuning is enough. I think there needs to be a physical two step process so the llm can read its own thoughts, giving it a chance to refine it. I have this coded and working in a fork of the llama 3 huggingface transformer but i got busy with my day job.

[-]

rambat1994@reddit (OP)

I agree, its kind of wild to think the entire foundation of this effort (and the hype) was basically fine-tuning on QA pairs. If anything this entire "model" could just be a implementation detail/process of an LLM. Kind of like doing anything with agents or making retrieval better for RAG

[-]

Mantr1d@reddit

i also built in a context window and a mini rag pipeline

[-]

Pro-editor-1105@reddit

something just doesn't look right about him.

[-]

Hunting-Succcubus@reddit

It’s confidence smirk.

[-]

Caffdy@reddit

Straight out of that website, thispersondoesntexist.com

But ended up being thismodeldoesntexist in this case lol

[-]

a_slay_nub@reddit

Can we stop giving him attention please?

[-]

punkpeye@reddit

I am out of the loop. What's good/bad about this guy?

[-]

Playful_Accident8990@reddit

He claimed to have created a state of the art model, which turned out to be him just making API calls to OpenAI's Chatgpt, then Anthropic's Claude. He said he'd release his "model" to the public, and it was just an old llama model from Meta.

[-]

punkpeye@reddit

I recall that. Wasn’t it a mistake? Some confusion that led him himself thinking he is talking to local model when requests were proxied OpenAI. Or at least I vaguely remember something like thag

[-]

Playful_Accident8990@reddit

He certainly claimed it was a mistake, each time others caught him on it

[-]

Hunting-Succcubus@reddit

I don’t know how i pulled trigger every time before victim died. It was mistake.

[-]

mikael110@reddit

No, that's just a theory one of his defenders posited. There was never any evidence of that (and plenty to suggest it's not true) and it's not a line Matt ever went with officially. Matt has never acknowledged that Claude or OpenAI models were used at all during any stage of this debacle.

[-]