I must have missed something, how is 2 bit working so well?

Posted by noneabove1182@reddit | LocalLLaMA | View on Reddit | 2 comments

I thought the sentiment was that going to 2 bit was going to result in WAY too much info and accuracy being lost, how are they performing so well now?

Reply to Post

2 Comments

[-]

NickCanCode@reddit

FYI, some guys are do pruning + quantization to speed things up: [https://neuralmagic.com/blog/speed-up-your-llms-with-sparsegpt-and-deepsparse-on-cpus/](https://neuralmagic.com/blog/speed-up-your-llms-with-sparsegpt-and-deepsparse-on-cpus/)

[-]

ArthurAardvark@reddit

Jeez. Every couple of months I stumble onto a thread here that just thunderclaps my face to remind me how fast this all moves. Had no idea about this + **variable quantization**. So if you're like me and thought 2B was still barely useable, 3 huge improvements.