The project can 100% be good for a final year, but it might be a little bit overdone, imo. Though when most people would just use LoRA to fine tune an llm and be done. You’re going a lot further than that. You’re gonna need to be comfy with PyTorch, jax or whatever have you, use as much prewritten code as you can, don’t get bogged down in writing like cuda kernels or smth. Oh and karpathy course and stuff, there’s so many tutorials. Though there are things you should look at beyond tutorials.

I would look at niche language llms on huggingface, you’d probably want to see if they reported hyperparameters or reported anything on weights and biases. Also, just cold contact their creators, the community is super nice.

Chinchilla scaling laws say about 20 tokens per param, albeit maybe outdated? and standard practice usually trains just with as many tokens as possible. Like 100s-1000s.

I’m not exactly sure how much code you want to write yourself, but you could try some smaller tweaks to the standard transformer model. You’ll be using the same prebuilt optimisers, maybe try moonshots one!

This will also depend on your course, could be computational linguistics to like straight maths idk. For the former more typical ngram models would be cooler, for the latter you’d probably want a more experimental.

Oh and also don’t underestimate how much a 24/7 gpu fan will drive you insane, there’s a reason my fine tuning rig is stuffed in a garage and I just SSH into it.