Calling a Finetune/LoRA Wizard: Need Dataset Tips for RP Model
Posted by AmpedHorizon@reddit | LocalLLaMA | View on Reddit | 17 comments
Hey everyone,
I've always wanted to do my own fine-tune/LoRA/QLoRA and I'm trying to get a better sense of the dataset size needed. The plan is to build a dataset in a specific style, but before committing time (and money), I'd really like to get a better sense of how to start properly without overshooting or undershooting.
Let's assume:
- We want to fine-tune a \~12B base model using a new clean dataset
- To make a general roleplay model, not tied to a single character, but with a certain structure
When we ignore the technical part and focus on creating the dataset in theory, for this kind of project, what's a good starting point? 30k examples in the dataset? More? Less?
If anyone has experience or resources they can share, that would be amazing (even rules of thumb). Or maybe a legendary finetuner around who can offer some guidance or practical tips on planning the dataset? If there's interest, I would also document my journey.
InnerSun@reddit
I'm not a finetuner but I've read up on a lot of stuff because I want to do some myself one day, and I think you might find a lot of ideas by searching what was already posted by the very first finetuners such as Teknium (NousResearch, Hermes), Migel Tissera (Tess/Synthia models), Eric Hartford (Dolphin) and the RP finetunes.
btw you can dig up all kind of "hidden" stuff using ChatGPT/Gemini/etc. search features as they index a lot of things.
From what I understand, 10k is ok as long as it's diverse enough. If it's anywhere close to Stable Diffusion LoRAs, if most of your examples are similar, it will converge to that style of answers.
There are a lot of datasets already available so you can go beyond 10k easily, and nowadays it's even easier to create one by transcribing videos, podcast, livestreams, OCR books, Reddit dumps, scrapping various forums, and so on.
The main challenge will be making sense of all this and reformatting it to the proper format that fits your model and the instructions structure you're going for.
AmpedHorizon@reddit (OP)
ty for sharing I'll read them. Regarding adiitional data and reformatting, do I really get a benefit in my case if I include them? The cool thing is, compared to some years ago, we now have really powerful models, were we can create all sorts of crazy synthetic data. Shouldn't it be enough to focus on creating a strong diverse synthetic dataset, for a fun RP model?
InnerSun@reddit
I think the main issue is that people fear they'll carry the bad GPTisms of the model (the overuse of metaphors, the way of speaking, abusive usage of emojis, etc.) into their finetune if they rely solely on synthetic data. It really depends on what style you want.
AmpedHorizon@reddit (OP)
that is really an issue, but I think we can counter it by using different models and samplers + putting in a lot of effort to refine it. I would love to experiment with this and dedicate time/effort to it.
InnerSun@reddit
I know the guys that worked on Dolphin and Tess basically milked every new API-only model on release to extract various datasets, so thats a strategy for sure
toothpastespiders@reddit
In my opinion the big problem there is that even the best cloud models tend to be pretty bad with creative writing. There's a strong tendency to just lean into whatever writing style is most associated with a subject. Like with pop-culture and claude it has a natural drift into writing like a stereotypical redditor. And that can often persist through prompting. Like claude trying to write like a 4chan user comes off as a redditor doing a bad impression of what he thinks 4chan is probably like. More caricature than mimicry. Positivity bias can seep in pretty easily as well. And when you're dealing with huge datasets it's really easy to miss something subtle like that with a quick skim.
Not saying it's not possible to get good results with synthetic data focused on creative writing. But it's a more challenging project than it seems like at first glance.
AmpedHorizon@reddit (OP)
I see the challenges you described and I share your concerns. And at the same time this sounds like a lot of fun in terms of experimentation! I believe in the tech and I think it is totally worth a shot with a clever multistep pipeline using different models. I totally hate the positivity bias and it would be number one on my list. But as much as I would love to just jump in, I need to do some planning on this and that is why it is crucial to me to learn more about the the dataset size.
Mbando@reddit
You can go away lower then 10k examples. If you review the LIMA paper, 1k to 2k high-quality and diverse examples are effective. I have personally gotten a high fantasy fine-tune on Mistral 7K using 650 high-quality diverse examples, both authors and tasks.
InnerSun@reddit
Interesting, looking at the big finetunes I always assumed you kinda needed a lot, but your example seems very similar to his project. Do you have a link to check out ? The dataset or the finetuned model itself.
AmpedHorizon@reddit (OP)
thanks I'll check out this paper. So your resulting model was able to reproduce your given structure and the setting? Did you ever feel that it reproduced too much content from the used dataset?
Mbando@reddit
I took a base model and got it instruction trained to write in Tolkien’s and Gene Wolfe’s styles, and write in first person, third description, and dialogue.
Not saying that low is ideal, just that quality and diversity are much more important than volume.
danielhanchen@reddit
If it helps, we added some finetuning tips and tricks to https://docs.unsloth.ai/get-started/fine-tuning-llms-guide/lora-hyperparameters-guide
AmpedHorizon@reddit (OP)
Thanks the whole https://docs.unsloth.ai/get-started/fine-tuning-llms-guide looks very valuable, I'll study it thoroughly before I begin creating the dataset (though I doubt that stated 1000+ rows are enough in my RP case, but it gives me confidence that 1000 rows have at least an impact on a base model)
toothpastespiders@reddit
I mostly train on non-fiction, so I'm not super familiar with chat datasets. But one small piece of advice. Make sure to be very careful about the quality. In my opinion a smaller amount of high quality data is far better than a large amount of mediocre data. And I think that's the trap that a lot of people fine tuning roleplay models fall into. The datasets I've glanced at there have often been in pretty rough shape. Likewise why I'd caution to be very careful with sourcing any pre-made datasets. There's tons of them out there. But not tons of high quality examples.
AutomataManifold@reddit
A lot of the finetuning discussion is going on in Discords, so one additional source of information is to track down the discords associated with various finetuners and ask there.
Hot-Employ-3399@reddit
Check PIPPA dataset and their paper
DecodeBytes@reddit
LoRA is data-efficient and usually needs 10×–50× less data than full fine-tuning.
I would say between 10-20k is about right, but it depends, sometimes less is more. It really depends though on what you're training. Are you trying to change the models knowledge, that is a bit more challenging, and can go quite wrong (catastrophic forgetting).
I would be curious in learning about how you plan to construct the dataset and maybe able to help curate it with / for you. I am currently working on https://www.deepfabric.dev and its always useful to see folks real world needs. If this sounds interesting drop me a PM.