So, that new sampling paradigm seems to be real after all

[-]

By investigating alternative top-k tokens instead of relying on greedy decoding, the authors show that inherent CoT reasoning paths can be found within LLMS, demonstrating their intrinsic ability to reason without prompting.

[-]

bearbarebere@reddit

I'm saving this comment in case it gets deleted for some reason because I have some research to do, thanks

"By investigating alternative top-k tokens instead of relying on greedy decoding, the authors show that inherent CoT reasoning paths can be found within LLMS, demonstrating their intrinsic ability to reason without prompting."

[-]

ObnoxiouslyVivid@reddit

I'm not sure if we can call different sampling paths "reasoning". It's the good ol' "regenerate it multiple times to see if you get a different answer" trick.

[-]

petrus4@reddit

Otherwise known as brute force search.

[-]

Kimononono@reddit

it’s search and a validation metric. Id say it’s analogous to being able to play an instrument well vs having a ear for good music.

This “confidence” generation method can’t improve inherent llm reason but can validate if a statement is reasonable. Both lead to the same outcome though.

[-]

Billy462@reddit

Any idea if this can be implemented on-top of an API or does the decoding really need to be done a lot deeper down?

[-]

Additional_Ad_7718@reddit

It would need to be implemented by the provider since it requires a modification of the inference code.

[-]

rileyphone@reddit

You can do your own sampling with a round trip for each token if you have prefilling and logprobs but it still doesn't really make sense. Gemini just shipped logprobs so someone might try to get it working (if they can manage to get API access that is).

[-]

schlammsuhler@reddit

So can we distill from gemini?

[-]

export_tank_harmful@reddit

So are we talking frontend (Sillytavern/etc), inference engine (llamacpp/etc), or actually within the model itself (via foundational training)?

Because only one of those three are out of our grasp...

[-]

youcef0w0@reddit

inference engine level, it doesn't require changing the model itself at all

[-]

Billy462@reddit

Makes sense, thanks for answering!

[-]

FuckShitFuck223@reddit

:0

[-]

gibs@reddit

They are not really the same thing. The paper cited is about exploring the search space of tokens other than the most probable, like with beam or tree search. Entropix is about using the logit distribution to figure out how to sample that token, and potentially inject a special token intended to provoke CoT.

Entropix at this point is still 100% hype 0% substance. No evals despite ample opportunity to have run them. Which is pretty sus imo.

Weirdly, their update says they are scaling up to a full production implementation -- without even having demonstrated that the thing works! It's just been cherrypicked single examples without any comparison to baseline.

[-]

Ok_Landscape_6819@reddit (OP)

From what I understand, entropix can resample or branch if a given token has too high entropy or varenthropy, which is very much so a toks space exploration akin to what's being researched in the paper. As for no evals, iI agree, until we see actual benchmarks it's nothing but air, but this paper suggest you can elicit COT from sampling alone which is a nice backbone for entropix claims.

[-]

gibs@reddit

"Elicit CoT from sampling alone" just means, if the model's most probably token is a direct answer like "5", the other tokens can be things like "Let's", "If", "Consider" etc., which lead to a CoT continuation rather than directly answering. It's not magic, it's just a way to explore the space of likely continuations.

[-]

kesor@reddit

https://notebooklm.google.com/notebook/d1022fe1-5c4c-42dd-993d-e6f8e8a19123/audio

[-]

remghoost7@reddit

Freaking love NotebookLM.

I made a repo with examples of how to turn it into an entirely different podcast, if anyone is interested.

I got it to use only a single narrator to make a fictional audio-drama named "Interesting Stories from the Void". Lovecraftian / Nightvale-ish sort of thing. Example in the repo.

[-]

Kat-@reddit

May i also recommend https://illuminate.google.com/library?play=cY1gap1_yh623

[-]

n0vella_@reddit

AI solving AI problems 😂

[-]

LiquidGunay@reddit

Sampling decisions need to be done using RL instead of heuristics and then we should get some really sweet results.

[-]

altoidsjedi@reddit

Hold on, this paper was published in Feb 2024? Updated in May 2024?

[-]

rileyphone@reddit

The paper is for a different technique involving decoding and a more basic confidence metric than varentropy. Entropix is still a greedy decoder, just changing the sampler (like min p or dynatemp).

[-]

Semi_Tech@reddit

So pretty much you create more possible answers(branches) and ascertain if the model feels more content with its own response?

Would this be the sauce behind o1?

When they say “Let the model think for longer” would that mean to have the CoT decoding use for example top 5-30 whenever a token is generated and see how certain a model is of the answer?

If that is the case the I can see how that is MUCH MORE compute intensive than greedy…

[-]

KillerX629@reddit

No, but may have some parallels, o1 uses some model for RL, probably a critic + one that keeps the "thought path"... We'll never know