LoLCATS - a hazyresearch Collection (of Linearized Llama 3.1 models 8B, 70B, and405B)

Posted by DeepWisdomGuy@reddit | LocalLLaMA | View on Reddit | 15 comments

[-]

cr0wburn@reddit

Even after reading the GitHub repository, I don't really understand what this does. Does it make llama 3 work better ?

Can anyone explain it like I'm 5 for me :)

[-]

i_wayyy_over_think@reddit

research paper and news article is here https://www.reddit.com/r/machinelearningnews/comments/1g3ra2m/stanford_researchers_propose_lolcats_a_cutting/

[-]

kryptkpr@reddit

It swaps quadratic attention for linear, which should improve inference performance at large context lengths especially if you don't have flash-attn.

[-]

FullOf_Bad_Ideas@reddit

MMLU his is pretty big.

83 >> 72.2 for Llama 405B

Can you think of any useful applications where a 405B model with a performance of a 20B model (70B non-linearized has 78.8 and 35B models have around 75-77 scores) but linearized would be practical?

Or a 70B linearized with a perfomance of 8B model.

[-]

You're looking at it wrong. If it's actually linearized then only a mmlu drop of that much in the conversion is fairly impressive. Linear attention on a sota model has a lot of potential in vram savings. That said it's unclear how to use these models? Also, pickles? Really?

[-]

kryptkpr@reddit

Inference code is here (pytorch): https://github.com/HazyResearch/lolcats/tree/lolcats-scaled/demos/

vLLM "soon"