10x longer contexts for reasoning training - 90% less memory GRPO in Unsloth

Posted by danielhanchen@reddit | LocalLLaMA | View on Reddit | 60 comments

Hey r/LocalLLaMA! Thanks so much for the support on our GRPO release 2 weeks ago! Today, we're excited to announce that you can now train your own reasoning model with just 5GB VRAM for Qwen2.5 (1.5B) - down from 7GB in the previous Unsloth release!

This is thanks to our newly derived Efficient GRPO algorithm which enables 10x longer context lengths while using 90% less VRAM vs. all other GRPO LoRA/QLoRA implementations, even those utilizing Flash Attention 2 (FA2).
With a GRPO setup using TRL + FA2, Llama 3.1 (8B) training at 20K context length demands 510.8G of VRAM. However, Unsloth’s 90% VRAM reduction brings the requirement down to just 54.3GB in the same setup.
We leverage our gradient checkpointing algorithm which we released a while ago. It smartly offloads intermediate activations to system RAM asynchronously whilst being only 1% slower. This shaves a whopping 372GB VRAM since we need num_generations = 8. We can reduce this memory usage even further through intermediate gradient accumulation.
We also implemented a highly memory efficient GRPO loss, which saves memory usage by 8x. Before 78GB was needed for 20K context length - now only 10GB!
Try our free GRPO notebook with 10x longer context: Llama 3.1 (8B) on Colab: https://colab.research.google.com/github/unslothai/notebooks/blob/main/nb/Llama3.1_(8B)-GRPO.ipynb

Blog for more details on the algorithm, the Maths behind GRPO, issues we found and more: https://unsloth.ai/blog/grpo

GRPO VRAM Breakdown:

Metric	Unsloth	TRL + FA2
Training Memory Cost (GB)	42GB	414GB
GRPO Memory Cost (GB)	9.8GB	78.3GB
Inference Cost (GB)	0GB	16GB
Inference KV Cache for 20K context (GB)	2.5GB	2.5GB
Total Memory Usage	54.3GB (90% less)	510.8GB

We also now provide full logging details for all reward functions now! Previously we only showed the total aggregated reward function itself.
You can now run and do inference with our 4-bit dynamic quants directly in vLLM.
Also we spent a lot of time on our Guide for everything on GRPO + reward functions/verifiers so would highly recommend you guys to read it: docs.unsloth.ai/basics/reasoning

Thank you guys once again for all the support it truly means so much to us! We also have a major release coming within the next few weeks which I know you guys have been waiting for - and we're also excited for it!!

[-]

kumonovel@reddit

Tasty and sweet! Thank you both and every other contributor for your tireless work!

But I also have a question. How much are VLM with GRPO like reasoning on your guy's roadmap?

ExaminationWise7052@reddit

Would it be possible to use a reasoning model to replace GRPO in the training of another reasoning model?

yoracale@reddit

They have different results but yes kind of. Just make sure your dataset has reasoning then

I wouldn't be incapable of doing it myself; I was just mentally rambling after reading your post, wondering if there was a way to make the weight adjustment that GRPO does in a smarter and more precise manner. Thank you for your response and for your work!

StyMaar@reddit

Can someone ELI5 why you need 510GB of VRAM to train an 8B model with 20k context?

How is this extra memory (around 490 more GB than for inference) being used?

Educational_Rent1059@reddit

Thank you for this!!

Thanks a lot for reading! 🙏

Ok_Warning2146@reddit

Thanks for your hard work. Will give it a try.

How's the progress on gemma support?

Thank you! Do You mean Gemma for GRPO or Paligemma in general? 🙏

I can confirm that gemma 2 GRPO is still not working with vllm using unsloth 2025.2.15

I want to run grpo with vllm on gemma

dahara111@reddit

It's timely! thank you!

I'm thinking of trying GRPO using unsloth/DeepSeek-R1-Distill-Qwen-14B, but are there any precautions, advice, or expected memory requirements?

It would be nice if it could be run on a single A100 (40GB) in the cloud.

For 14B QLoRA youll need just like 20GB VRAM so 40GB VRAM will be fanastic!

Just make sure you get the reward function/verifier right

So it's possible to test it on the 3090, thank you

I've been able to test it on the 3B.

But I felt I needed a bigger model with a longer context to get the model to do something more useful. That's very timely and I'm grateful, I'll share it if it works. Thanks !

Yes correct! I think smaller or bigger model doesnt matter. A bigger model will help you reach aha moment faster but then with the smaller model, training is much faster so u can train it for longer

You_Wen_AzzHu@reddit

God bless DanielHanChen.

danielhanchen@reddit (OP)

Appreciate the support as always!

pm_me_ur_sadness_@reddit

Hi bro i saw the job apps on X to join unsloth

Is anything similar open for interns

Hey yes there is. We also have set bounties for solving Github so your contributions feel definitely valued :)

Robo_Ranger@reddit

Thanks for your hard work! I read your docs and noticed that you mentioned, "The best part of GRPO is that you don't even need that much data." Could you tell me the minimum data size required for effective training?

Thank you! Maybe like 100 rows. Can be even less but will take the model longer to train

CheatCodesOfLife@reddit

Took me longer than it should have to realize the link is broken / wonder why I was getting github authorization issues lol

Correct link: https://colab.research.google.com/github/unslothai/notebooks/blob/main/nb/Llama3.1_(8B)-GRPO.ipynb

Whoops much apologies, isn't that the same link as we linked?

Maybe old.reddit.com formatting, I see it as

Try our free GRPO notebook with 10x longer context: Llama 3.1 (8B) on Colab: https://colab.research.google.com/github/unslothai/notebooks/blob/main/nb/Llama3.1_(8B)-GRPO.ipynb-GRPO.ipynb)

And clicking the link tries to open: https://colab.research.google.com/github/unslothai/notebooks/blob/main/nb/Llama3.1_(8B

Oh crap you might be right thats very weird

AD7GD@reddit

Formatting is new/old reddit dependent. For example >!spoilers!< >! don't work !< if there are spaces, but only on old reddit.

az226@reddit

Can you get the same/similar benefits in pretraining and/or full parameter fine tuning as well?

Yes ofc but its not supported in unsloth atm. hopefully very soon

lumierenoir@reddit

As always, the unsloth delivers improvements for us GPU poors. Thanks for all of your hard work to make this possible

Appreciate the kind words!!

throwaway2676@reddit

Someone needs to think of the poors!

Thank you thank you for the support! :)

MLDataScientist@reddit

Thank you! Amazing progress!

Question: does unsloth library support full training of the model (e.g. SFT)? or LoRa fine-tuning for now?

Is multi-GPU now supported?

unsloth is a great library. Thank you again!

We'll be releasing some stuff everyone has been waiting for in the next few weeks!!!!

Uncle_Warlock@reddit

Multi-GPU support, even if it's only for 2 or 4 GPUs (I have two 3090s with NVLink, and 128GB of DDR4 RAM), would be a really huge help! Thank you for all you do!! 😊

Thank you!! I'm working around the clock to get it done!!

FancyMetal@reddit

This came in timely, I have a small project(hf.co/Lyte/QuadConnect2.5-0.5B-GRPO) to be done and chose to make an LLM reason over a game(Connect 4) I just started today and opened one of the notebooks from github, huge thanks for all the amazing resources, god bless you all at unsloth!
Seeing the reasoning emerge from a small 0.5B model without being forced is actually so exciting!

Thank you that's amazing! Thoguh we'd mostly recommend people to start from models with at least 1.5B parameters as sometimes the models won't be able to work

yes, I was surprised because I heard that models smaller than 1-1.5B can't learn to "reason"(well at least learn to think) but it surprisingly did which was fun!

Yes! I was very surprised small models still learn reasonably well if given good reward functions and enough time for compute!

Thrumpwart@reddit

Does this support AMD ROCm?

Does this support FP16 training? Or only quantized versions?

At the moment no.

Yes, you can do LoRA or QLoRA with this

Thank you! Looks awesome nonetheless.

Thank you!!

TyraVex@reddit

Hi, thank you for this! I have 3*24gb vram, is unsloth still limited to a single GPU?

Currently yes but multiGPU is coming and in the works!

Can't wait! Thank you for all the good work!

nite2k@reddit

for now, yes but it's on the roadmap

IcyBricker@reddit

Do you have other reward functions? The reward functions all seemed geared towards math or matching phrases.

What if you just want a reward function that rewards base on how different it is in creating something completely new and innovative.

TheRealMasonMac@reddit

This is my idea: https://www.reddit.com/r/LocalLLaMA/comments/1iszn3e/can_you_grpo_train_a_model_for_open_ended/

Yes absolutely that can be a reward function, we wrote a lot more about it in our guide: https://docs.unsloth.ai/basics/reasoning-grpo-and-rl

Were there issues with multigpu? I recall reading that you ran closed testing on it a few months ago.

Yes, we allowed community members early access to multiGPU. Hopefully we'll release it real soon!

Also for those using notebooks, I managed to make all the rewards show up!