Our 3rd AMA: Unsloth Team, Creators of the lightning-fast Unsloth fine-tuning library! (Wednesday, 10 AM-1 PM PST)

I have a question I've really never seen addressed well in all of the many fine-tuning videos, blogs, articles, etc. as most of them focus on training LLMs to respond to chats or instructions in a certain style or format. At our work we use a specialized piece of software which is similar to VB but highly customized to the point where even a coding LLM that was trained on VB would still get things wrong. I have plenty of code examples as well as the developer documentation which is highly-detailed and definitely contains everything one would need to know in order to properly script something. I understand the concepts of fine tuning and have done it plenty of times with text and image based models, but when it comes to training a coding LLM I get stuck. If you know of any good resources that go into greater detail on how best to do this I'd love to know about them. Perhaps you might even consider creating a fine-tuning notebook or blog article specifically about best practices for training a coding model. Ideally, I'd like to have a model (or two, depending on suggestions) that can both generate code (input the requirements, get code out) as well as something that can be used conversationally to answer questions about the language, suggest code improvements, help correct errors in code, etc. Some of the things that I get stuck on: * Should I train a base model first to let it 'learn the patterns' of the language, then do instruction tuning for generating code and answering questions, or is the current state of models / fine-tuning sufficient to where I can skip straight to an existing instruction-trained coding model (perhaps one already trained on VB)? * Between documentation, code examples, archived conversations between developers discussing the software and scripting concepts (email, forum posts) and synthetically generated Q&A or instructions/outputs, roughly how much of each should there be in the training data? * How should chunking be approached with code? Even with some of the content I've found specifically about creating training data for coding LLMs, it's for languages which are easily split into multiple files and thus an entire file can fit into the context window. In the case of my custom scripting language, all code for a particular use case must be contained in a single file and can get quite large. If I have example code that's too long for the model's context window, do I simply throw it out? Cut out what I can so that it still remains valid? Simply truncate the file and add an indicator at the cut points that it's continued from elsewhere? * When it comes to fine-tuning coding LLMs, how much training data should I aim for? (I suppose this might differ based on whether I'm using a model which is already familiar with VB vs one only trained for the usual languages, Python, HTML/CSS/JS etc) * Any model suggestions for my use case? I started down this road back when the first major Llama model came out and when Unsloth first came on the scene - I've been wanting to give it another shot with some of the newer models out there but it seems like if you stop paying attention to the space for a week you're already out of date! I know I asked a lot of questions - any guidance you can provide on any of these points would be a tremendous help! Thanks in advance and thanks for all the work you've done for the community.