Mistral Nemo is really good... But ignores simple instructions?

Posted by Majestical-psyche@reddit | LocalLLaMA | View on Reddit | 24 comments

I've been playing around with Nemo fine-tunes (magnum-12b-v2.5-kto, NemoReRemix) They're all good really good at creative writing, but sometimes they completely ignore 'simple' instructions... I have a story with lots of dialog, and I tell it to: Do not write in dialog. But it insists on writing with dialog; completely ignoring the instructions. \_\_\_ I tried a bunch of chat templets (Mistral, ChatML, Alpaca, Vicuna)... None of them worked. Settings: Temp: 1, Reptation penalty: disabled, DRY: 1/2/2, Min-P: 0.05. \_\_\_ Anyone have advice - tips for Mistal Nemo? Thank you 🙏❤️

Reply to Post

24 Comments

[-]

grimjim@reddit

I've resorted to merging creative fine-tunes with the OG Nemo 12B instruct to get steerability back. I've been sticking with (my own) grimjim/Magnolia-v3-12B for its steerability recently, though grimjim/Magnolia-v4-12B is somewhat more creative without prompting to promote creativity. It's not perfect though. But in short, what you want sounds like instruction-following.

[-]

ProcurandoNemo2@reddit

I don't know about any of the finetunes, but I've been using the original instruct version and, at 20k context for a book I'm writing, it's still very consistent at following my prompts. Coupled with the more natural writing style, it has replaced Command R+ for creative writing for me.

[-]

my_name_isnt_clever@reddit

I'm curious, what front end do you use for creative writing?

[-]

ProcurandoNemo2@reddit

Text Generation WebUI by Oobabooga. It's not perfect (I'd rather use something more like Novelcrafter) but it helps. At least the answers can be edited.

[-]

Nice_Grapefruit_7850@reddit

Can they not be edited in novel crafter? I was thinking of switching over to it but if I can't edit the ai's response then it's pretty useless.

[-]

LoafyLemon@reddit

I've had the same experience as you after trying out both the magnum KTO, and NemoReRemix. My recommendation would be to use standard Instruct or RP/Prose, and Sao10k's Lyra v1 for ERP. Why? Because it seems the limited ERP data in the standard instruct fine tune makes it loop during ERP, otherwise it's absolutely fine. Lyra model however is a little dumber and limited to 16k context, but overall is more creative.

[-]

Technical-History104@reddit

Any “don’t do” type instructions will be weaker, even on the best models. You either need to go the lengths of describing what you “do” want, or try another way of expressing what you “don’t” want without using a simple negation.

[-]

wakigatameth@reddit

Pure Mistral Nemo instruct is much better at complying with instructions than magnum 2.5 kto or Nemo Remix.

[-]

daHaus@reddit

Using a whole number for temp makes it a crap shoot, try with .99 or the default (.3) and see if it behaves differently

[-]

jungle@reddit

I'm trying to get it to summarize a podcast transcript (the latest TWIT episode - 3 hours long, 30757 words), but for some reason I can't get it to summarize more than the few last minutes of it, even if I explicitly set the 'num_ctx' parameter to 131072.

[-]

daHaus@reddit

Oh yeah, you'll definitely want a low temp for that to suppress hallucinations. .3 or less is probably best for that. Is that ollama that uses num\_ctx? I'm not familiar with ollama but the RAM needed for +131 k context would be extreme. Does it work with 32,768 and 65,556 context? It may be running out of memory and silently failing.

[-]

jungle@reddit

Yes, ollama. You make a good point. I'll see if it has a DEBUG logging option or something. *: Turns out it's truncating the input: > INFO [update_slots] input truncated | n_ctx=2048 n_erase=36946 n_keep=4 n_left=2044 n_shift=1022 tid="0x1fdf67ac0" timestamp=1723982237 I'll see what can be done about it. I know it can take more than 2K tokens.

[-]

jungle@reddit

Yes, ollama. You make a good point. I'll see if it has a DEBUG logging option or something.

[-]

f3llowtraveler@reddit

If you want to tell it not to use dialogue then don't spell it as dialog.

[-]

CheatCodesOfLife@reddit

Try the original Nemo-Instruct.

[-]

jungle@reddit

Isn't that a finetune? All the advice points against using finetunes...

[-]

ArtyfacialIntelagent@reddit

> I've been playing around with Nemo fine-tunes ... but sometimes they completely ignore 'simple' instructions... Yes they do. About 90-95% of finetunes are complete crap. They're crap because curating good datasets takes a LOT of work, way too much for small-scale model trainers. When you finetune on mediocre data, you get something roughly similar to the original model but much more stupid. > Anyone have advice - tips for Mistral Nemo? Yes. Avoid the lobotomized finetunes. Use the Nemo official release. It's incredibly good for its size. It doesn't even need uncensoring if you write your prompts carefully.

[-]

GoogleOpenLetter@reddit

> It doesn't even need uncensoring if you write your prompts carefully. "In the past....[ ]"

[-]

-Ellary-@reddit

You using creative finetuned models, they will "do" their thing no matter what, they are biased. Try to use official inst. model of Nemo, but if you need model with better instruction following use Gemma2 9b. You cant have it all on a small models, there is always be something, but HEY you can use Mistral Large 2 complete free at mistral website.

[-]

Dead_Internet_Theory@reddit

Do consider if the chat templates are not in contradiction with what you're asking. E.g., "you are {{char}} in this conversation with {{user}}" (or something like that) is basically asking it to write a dialogue. If you already tried editing that - try messing with the samplers, sometimes that help.

[-]

Red_Redditor_Reddit@reddit

If you mean the temp is 1.0, I think you've set it too high. I don't know how that translates to it ignoring instructions, but you might want to set it to a lower number like 0.3. You might also specify in the system prompt that it should not respond in story dialog. I might even say in the system prompt "you are a system that helps user write stories. Do not write responses as part of the story." That way it doesn't get confused as to its role in your dialog.

[-]

Mistral Nemo is really good... But ignores simple instructions?

Reply to Post

24 Comments

grimjim@reddit

ProcurandoNemo2@reddit

my_name_isnt_clever@reddit

ProcurandoNemo2@reddit

Nice_Grapefruit_7850@reddit

LoafyLemon@reddit

Technical-History104@reddit

wakigatameth@reddit

daHaus@reddit

jungle@reddit

daHaus@reddit

jungle@reddit

jungle@reddit

f3llowtraveler@reddit

CheatCodesOfLife@reddit

jungle@reddit

ArtyfacialIntelagent@reddit

GoogleOpenLetter@reddit

-Ellary-@reddit

Dead_Internet_Theory@reddit

Red_Redditor_Reddit@reddit

grimjim@reddit

Kep0a@reddit

Xhehab_@reddit