Hosting Assistant_Pepe_70B on Horde!

Posted by Sicarius_The_First@reddit | LocalLLaMA | View on Reddit | 5 comments

Hi all, Hosting [https://huggingface.co/SicariusSicariiStuff/Assistant\_Pepe\_70B](https://huggingface.co/SicariusSicariiStuff/Assistant_Pepe_70B) on Horde at very high availability on 2xA6000. FP8 precision at 16k context (FP8 is about 99.99% accuracy). ( [https://lite.koboldai.net/](https://lite.koboldai.net/) FREE, no login required) So give it a try! (Feedback always welcomed)

Reply to Post

5 Comments

[-]

Alan_Silva_TI@reddit

If you can give it a GGUF Q4K_M people will give it more attention. I'm very interested in using the model, but is very hard to run it ATM.

[-]

insulaTropicalis@reddit

[https://huggingface.co/SicariusSicariiStuff/Assistant\_Pepe\_70B\_GGUF/tree/main](https://huggingface.co/SicariusSicariiStuff/Assistant_Pepe_70B_GGUF/tree/main) Here you go.

[-]

Alan_Silva_TI@reddit

Thank you!

[-]

toothpastespiders@reddit

Nice! I've been putting off downloading until I cleared some drive space and decided just how much cope I need for a quant. I really think that training on datasets like this which highlight multiple viewpoints on any given issue and have actual disagreements and longer arguments built in are an underappreciated element of a solid dataset. Especially as models become more sychophantic.

[-]

Narrow-Belt-5030@reddit

It's actually pretty damn good. Shame won't fit on a 5090.