GPT2 using MLX

Posted by Disastrous-Maybe2501@reddit | LocalLLaMA | View on Reddit | 3 comments

Hi all, I was learning LLM pre-training from Andrej Karpathy's NanoGPT and decided to try it out using MLX. I originally thought it would be more or less a simple translation from PyTorch to MLX, but it turned out to be much more tricky than that. I published my code and documented my learnings in a blog post included in the repo. I'll kick off full training on fineweb on my M3 Max and will be publishing the training results to the repo once I have that. Any thoughts and feedback are welcome, here or directly on the repo. Thanks!