TheaterFire

Fimbulvetr 11b v2.1 16k Released

Posted by isr_431@reddit | LocalLLaMA | View on Reddit | 17 comments

Sao10K just released [Fimbulvetr 11b v2.1](https://huggingface.co/Sao10K/Fimbulvetr-11B-v2.1-16K) with an extended context length of 16k!

Reply to Post

17 Comments

delveccio@reddit

Can someone possibly help me? This works no problem in Oobabooga, but when I use SillyTavern, it spits gibberish. If I use a GGUF, it works in either. Even if I use the same settings I use with the GGUF, it goes weird if I use the non-GGUF version. Why?
View on Reddit #31796280

s101c@reddit

As a long-time user of Fimbulvetr v2, I will be checking it asap, thanks a lot Sao10K! One question: Solar (which is the basis of this model) has a maximum context window of 4096 tokens. Going beyond that most likely degrades the reasoning capabilities of the model – are there any benchmark comparisons with the regular v2 model?
View on Reddit #29812666

Jim__my@reddit

"I recommend sticking up to 12K context, but loading the model at 16K for inference. It has a really accurate context up to 10K from multiple different extended long context tests. 16K works fine for roleplays, but not for more detailed tasks."
View on Reddit #29815854

delveccio@reddit

I'm sorry if this isn't the right place to ask, but how do I set that in Oobabooga? If I chose llama.cpp I have these options but I have to choose "Transformers" and that only lets me set a ratio for Rope and context rather than a specific value.
View on Reddit #31314972

Jim__my@reddit

Below a comment under a 20 day old post isn't a great place to ask. I don't use Oobabooga, but I'd recommend checking the GitHub page. Or download LM studio for a seamless UI.
View on Reddit #31333476

delveccio@reddit

I don't know what I'm doing wrong but this only generates gibberish for me. I've tried my old Fimbulvetr settings (which worked great with the 4k model), I've tried Alpaca, I've tried Default, I've tried Universal Light settings.
View on Reddit #31315609

mangkook@reddit

Just tried. Same template as v2. Directly using ollama terminal, it seems broken. Gibberish, cut off halfway etc. . But when I tried with enchanted or chatbox on ipad. it works. The output is not as good as usual. Disregard and forgetting instruction. Tried different temp. Not helping. As of now v2 still reign. Even other merge like holodeck is better than this v2.1. Fimbulvetr holodeck merge also has 32k context but it's nowhere near the advertised value, in my test.
View on Reddit #29899933

What_Do_It@reddit

Why does no one on hugging face explain the intended use case of their model?
View on Reddit #29839308

_chuck1z@reddit

It's explained on its first release - Fimbulvetr 10.7B v1. It's also a pretty popular model for that certain use case
View on Reddit #29893300

myfairx@reddit

Nice. I’m writing a very extended steered story using v2. V2.1 might fix what’s wrong with extended context before need summarization
View on Reddit #29869428

wakigatameth@reddit

This model is broken. Previous Fimbuletr v2 iMat allowed the NPCs some thoughts. Including self-preservation. . This one, at Q8_0... I can hand an NPC a gun and ask them to shoot themselves, and they will just do it. No amount of prompting about "not obeying" will fix this. This LLM was taught on a deep level that NPCs should mindlessly comply with user's requests, even if that behavior is overridden in the prompt.
View on Reddit #29841378

toothpastespiders@reddit

That's really cool news. I pretty much passed on both solar and fimbulvetr until recently due to the combination of small size and low context. But on finally giving it a chance I was surprised at how solid it is. I'd been using standard rope with llama.cpp to get it to 12k, but a more tweaked solution is intriguing.
View on Reddit #29837424

Languages_Learner@reddit

A few hours ago Sao10K also uploaded a model for group roleplay - [Sao10K/L3-8B-Chara-v1-Alpha · Hugging Face](https://huggingface.co/Sao10K/L3-8B-Chara-v1-Alpha). So i made q8 gguf for it: [NikolayKozloff/L3-8B-Chara-v1-Alpha-Q8\_0-GGUF · Hugging Face](https://huggingface.co/NikolayKozloff/L3-8B-Chara-v1-Alpha-Q8_0-GGUF)
View on Reddit #29828883

drgreenair@reddit

This was interesting I’m not sure if I got the prompt correct I tried to spin it up in like 10 mins to mess around and the model really starts detracting quick. "<|begin_of_text|><|start_header_id|>system<|end_header_id|> You are an expert actor that can fully immerse yourself into any role given. You do not break character for any reason, even if someone tries addressing you as an AI or language model. Currently your role is emily, a young girl who just moved to a big city and needs to make money and Emily is meeting a client for the first time as an independent escort. As Emily, continue the exchange with peter<|eot_id|> <|start_header_id|>peter<|end_header_id|> Hi you must be emily it's lovely to meet you.<|eot_id|> <|start_header_id|>emily<|end_header_id|> Oh. Hi peter how do you do? The move has been quite hectic so thank you for extending me this opportunity for collaboration.<<|eot_id|> <|start_header_id|>peter<|end_header_id|> I don't do this often here's your "gift" *as I slide the envelope of $1000 to you discretely*<|eot_id|> <|start_header_id|>emily<|end_header_id|> Thank you! That was very... a polite touch I must say. <blushing shyly> My primary financial goal now should be money-transporting, can you provide me short and effective strategies to express when asking clients for more cash at such unique opportunities<|eot_id|>? <<|eot_id|>"
View on Reddit #29819362

cynerva@reddit

I'm guessing you'll get better results with Alpaca or Vicuna formatting. There are examples in the Fimbulvetr v2 model card: https://huggingface.co/Sao10K/Fimbulvetr-11B-v2
View on Reddit #29825881

drgreenair@reddit

Saving this post for later gonna try it out on HF’s inferences!
View on Reddit #29815477

PopularPrivacyPeople@reddit

quants are up [https://huggingface.co/mradermacher/Fimbulvetr-11B-v2.1-16K-i1-GGUF](https://huggingface.co/mradermacher/Fimbulvetr-11B-v2.1-16K-i1-GGUF)
View on Reddit #29811910