I wasn't expecting anyone to notice this model, I was going to make a post about it this weekend.

I created several synthetic datasets from scratch including a lot of long context multi-turn chat from chub.ai characters (and other places I pulled from). I generated as many turns as would fit in the 4k output window, prompting the model (command-r-plus) to create both sides of the conversation in one shot. My rationale was, by doing many turns in one generation, the model would be able to create a more coherent, naturally flowing conversation. Also, it's much more tokens per second from my experiments, and I only have 6gpu to play with so that was a factor. Once I got the first 8kish tokens of chat history finished, I let abliterated mistral-small-instruct continue batching the chats. Once the model had 8k example context for the conversation, I didn't see any quality decrease from switching from command-r-plus to the faster model.

I also used some existing datasets for story writing instruct, and I used a few categories from airoboros 3.2, mostly for non erotica "de-alignment" (re-alignment?). I ended up with something like 1.3gb of data total. I ran all that through abliterated Mistral-small-instruct, scoring it for quality. With repetition being penalized severely, and underage sexual content given an automatic 0. That took me down to about 900mb of data.

Ran that through axolotl with a lowish learning rate, only one epoch, and some parameters I pulled out of my ass. My rationale was, I'm not really trying to teach the model any new knowledge. My main goal was to get long context multiturn working as reliably as possible, without going off the rails, or resorting to loops or repetition. With as little gpt-isms and slop as possible. I remember reading, that the more you try to "teach" a model through fine tuning, the more of the "intelligence" of the base model is damaged, especially for tasks not relatable to training data. Citation needed. So I thought it made sense to try to teach long context and multiturn over a big range of samples with a light touch, basically trying to convert the base model to long context multi turn instruct model while damaging the original model as little as possible.

I'm launching a service that provides managed solutions for NSFW generative AI software. You pay a monthly fee, and get access to a web instance hooked up to my backend, pre-configured. Some example: SillyTavern, Agnaistic, OpenWebui, LibreChat, with stable diffusion and voice services plugged in. So I wanted to create a model that would excel for this specific environment. The most urgent feature was stable multiturn, because most models seem to fail this badly. But I think also there could be some big improvements with things like training data of common RAG solutions (ex, SillyTavern lorebooks) and inline/native stable diffusion prompt generation for each software, among other things. The models and datasets will be open source, because open source makes my dick hard. Plus, I feel like my target audience are the kind of people who just want to try/use this kind of software for fun without spending a bunch of money on hardware and learning about LLMs. They just want to sex the computer. I just need a few more patents to expire and then we can even have AI controlled sex toys. SillyTavern Orange Pi robo dildo/fleshlite plugin TBA.

I will make my own post trying to expand on some of this and get community feedback this weekend.