How much VRAM needed for Qwen 3.6 27B Q8 with 262K context?

[-]

chrismakingbread@reddit

I have 2x5090 (so 64GB TP=2) and I’ve managed to get to 218k context at 80-100 TPS with Qwen 3.6 27B FP8. Using FP8 model weights and KV with MTP=2. I think if you really need the extra context you’re going to want to target more like 96 GB.

Reply

[-]

ohhi23021@reddit

how? i get that on two 3090s with Q8 full KV16..... i guess fp8 is just giant.

Reply

[-]

TheSlateGray@reddit

I actually have a 48GB card. Just loaded Unlsoth Q8 with BF16 mmproj, F16 cache. CUDA0 : NVIDIA RTX PRO 5000 Blackwell (48439 MiB, 47628 MiB free) 30% 37C P8 17W / 300W | 46154MiB / 48935MiB

Reply

[-]

My_Unbiased_Opinion@reddit (OP)

this is without mtp right? are you over 48gb with mtp?

Reply

[-]

ohhi23021@reddit

i can do 230k WITH MTP (no video) on dual 3090s, if quantized the V to Q8, that should hit 256k... but i have it at 200k full quant, i don't think the extra 56k will amount to much.

Reply

[-]

TheSlateGray@reddit

Without MTP. My speeds are good enough to not want to sacrifice context length or quality for it. I usually run with no mmproj to save a little VRAM also, just enabled it so I could copy&paste some real numbers for you.

Reply

[-]

My_Unbiased_Opinion@reddit (OP)

thank you I appreciate it.

Reply

[-]

alexp702@reddit

From memory it’s about 53gb - LM Studio has a little slider that shows memory requirements pretty accurately.

Reply

[-]

ohhi23021@reddit

i use 200k context with Q8 on dual 3090s.... MTP too and 3gb is still left, full KV quant, 90 t/s and 70 t/s at 120k context....... i could probably do some llama.cpp updates and get 256k by making V q8 instead of q16....

Reply

[-]

TheSlateGray@reddit

I moved off LM Studio for straight llama.cpp because it would always says 27b needed much more VRAM then it truly needed for the context. Just updated the AppImage now, and it's showing I need 77.86 GB to load the same settings I use in llama.cpp, not sure what I'm doing wrong with LM Studio, but it's not perfect for sure.

Reply

[-]

FakeFramesEnjoyer@reddit

That memory estimate is janky and it says "beta" next to it for a reason, its a rough estimate and when you're loading in cutting edge models LM studio is always going to lag behind a bit in general. Not that it matters for your use case, since you already switched to raw llama.ccp which is a lot more versatile, but people should know LM Studio has a "model loading guardrail" option that can sometimes cause crashes if its set too sensitive. If you set that to OFF, and just ignore the the VRAM estimation, it should load in whatever model you want no problem (provided you're really not trying to load something in that doesn't fit).

Reply

[-]

idkanythingabout@reddit

I think it even says it's a an estimate in app if i recall

Reply

[-]

GabrielCliseru@reddit

and depending on GPUs the OP needs a bit more for communication overhead. I found 2xR9700 crashing from time to time. RTX5000 with 74gb does not crash

Reply

[-]

Evgeny_19@reddit

I think that it's just something is wrong with llama.cpp for multi GPU setups (at least for R9700s). When I use an old build from about two weeks ago it works fine for me on dual R9700s. And the mtp performance increase is much more pronounced that on recent llama.cpp builds (official container builds form ggml-org). Old build works fine with full 262k context. Although the model's quality seems to be degrading anyway, when the context goes beyond 130k. Even with f16 cache.

Reply

[-]

farcryjohn@reddit

I'm glad to hear someone else is experiencing the same thing. I thought I was going crazy. I also have dual R9700s, and was running into crashes with even just the Q6_K_XL from unsloth. Q_8 was even worse. I never used to have issues until the MTP update. I'm on LM Studio though, so I assumed it was something wonky with that. It actually caused a kernel panic several times before I just dropped back to Q4_K_M. It's weird, because using nvtop, I never saw the graphics memory get that high. It would happen during inference though, so I was also thinking it was the checkpoints taking too much system memory. I increased my swap partition and still ran into it though. No crashes on Q4_K_M though. ¯\_(ツ)_/¯

Reply

[-]

luncheroo@reddit

I've been looking at 9700s because I have reached a ceiling with smaller models and my work that I can't scaffold my way out of. Am I reading you both right that you prefer recent llama.cpp builds through LM Studio over vulkan?

Reply

[-]

farcryjohn@reddit

LM Studio uses llama.cpp as the backend, just with pluggable builds for MLX/Vulkan/ROCm/CUDA. I use Vulkan with both native llama.cpp, and LM Studio. I've tried running both LM Studio and llama.cpp. I haven't noticed a huge difference, other than the fact that you get a lot more customizability with llama.cpp. The defaults on LM Studio are typically good enough though, and the automatic model switching with LM Studio is extremely convenient. That's honestly the only reason I'm using LM Studio over llama.cpp.

Reply

[-]

luncheroo@reddit

Thank you for that. Are you getting everything you want with the vulkan as-is in LM Studio? I'm just curious about the amount of fiddling before I dive in. I've been wary of ROCm based on what I've read thus far, but I also understand that it's come a long way. If LM Studio + vulkan works pretty well out of the box, I'm sold.

Reply

[-]

farcryjohn@reddit

Vulkan has been working great for me using LM Studio / llama.cpp. I have had some minor issues with getting vLLM to run efficiently, but I have been able to get it to run using AWQ-INT4 quants. Ultimately, the prompt processing didn't seem to benefit from tensor parallelism, so I haven't been using it. With vLLM, I was just trying to get better parallelism than llama.cpp. I've also had some issues with certain tools, like Qwen3-TTS having poor performance. There's a few rewrites to get better RTF (Real Time Factors) that are CUDA specific. In addition to LLM workloads, I also run ComfyUI with no problems as well. I've done video generation with LTX and WAN, and no issues there either. Overall, I'd totally recommend the dual R9700s. If qualitative evidence isn't enough, I've posted a benchmark I ran with llama-benchy. X870E Taichi AMD Ryzen 7 9800X3D (16) @ 5.27 GHz AMD Radeon AI PRO R9700 32GB AMD Radeon AI PRO R9700 32GB 60.40 GiB |model|test|t/s|peak t/s|ttfr (ms)|est\_ppt (ms)|e2e\_ttft (ms)| |:-|:-|:-|:-|:-|:-|:-| |qwen3.6-27b@q4\_k\_m|pp8192|1431.98 ± 0.00||5292.12 ± 0.00|5281.49 ± 0.00|5292.12 ± 0.00| |qwen3.6-27b@q4\_k\_m|tg64|28.16 ± 0.00|35.00 ± 0.00|||| |qwen3.6-27b@q4\_k\_m|tg128|28.76 ± 0.00|36.00 ± 0.00|||| |qwen3.6-27b@q4\_k\_m|pp16386|1510.30 ± 0.00||9734.52 ± 0.00|9723.89 ± 0.00|9734.52 ± 0.00| |qwen3.6-27b@q4\_k\_m|tg64|28.28 ± 0.00|35.00 ± 0.00|||| |qwen3.6-27b@q4\_k\_m|tg128|27.96 ± 0.00|34.00 ± 0.00|||| |qwen3.6-27b@q4\_k\_m|pp32768|1521.63 ± 0.00||19634.26 ± 0.00|19623.63 ± 0.00|19634.26 ± 0.00| |qwen3.6-27b@q4\_k\_m|tg64|28.10 ± 0.00|34.00 ± 0.00|||| |qwen3.6-27b@q4\_k\_m|tg128|27.05 ± 0.00|34.00 ± 0.00|||| |qwen3.6-27b@q4\_k\_m|pp65535|1429.05 ± 0.00||41736.43 ± 0.00|41724.78 ± 0.00|41736.43 ± 0.00| |qwen3.6-27b@q4\_k\_m|tg64|24.99 ± 0.00|32.00 ± 0.00|||| |qwen3.6-27b@q4\_k\_m|tg128|24.33 ± 0.00|32.00 ± 0.00||||

Reply

[-]

luncheroo@reddit

This is great, thank you!

Reply

[-]

My_Unbiased_Opinion@reddit (OP)

yeah im using lmstudio as well. I find it easy af to use and is way better than ollama. (I mean anything is better than ollama lol)

Reply

[-]

DiscipleofDeceit666@reddit

I did notice that! The first official mtp build gave me blazing high speeds and the later builds gave me a performance penalty of like 10-15% when compared.

Reply

[-]

KeySea5392@reddit

I'm too poor for this thread

Reply

[-]

CodeDominator@reddit

Yeah. I have a B60 with 24GB RAM and I found out quick that I need at least another one to comfortably work with the 27B. Unfortunately this case is pretty binary - you either have enough VRAM or you don't. I'd say 32GB is minimum requirement for the 27B to be able to run Q6 with 128K context.

Reply

[-]

iijei@reddit

me too.

Reply

[-]

KeySea5392@reddit

I'm running tiny Q4 models on my ancient 8gb GPU 😓. I can't fathom what these people are doing

Reply

[-]

johnnydotexe@reddit

It's not much better over here in the 16gb camp.

Reply

[-]

KeySea5392@reddit

I don't understand why I need to buy a whole RTX card that does a million things I don't need. Just give me cuda/tensor cores and vram cards

Reply

[-]

johnnydotexe@reddit

There are more 'dedicated' cards and boxes for AI use, but those are expensive too. The vram chip costs and shortages killed the market. Can't even build a new gaming rig without having to spend a grand or more on just the ram and SSDs. Want a 5090 in it? $4k if you can find one in stock. It's going to be like this for the foreseeable future.

Reply

[-]

iijei@reddit

i have 12gb 3060, upgraded from rx 580. wish I can also get 48gb.

Reply

[-]

KeySea5392@reddit

I use Collab from time to time for small projects, then at least I get to use their 16gb GPU. But downloading the models each time is a pain

Reply

[-]

Sofakingwetoddead@reddit

You can fit q8 with full kv depth and max context at 47.5gb . There is a thread around here with it running cleanly on a single RTX 5000 PRO.... For me on SGLang, it's \~87gb with my settings but I can serve parallel requests that actually run in parallel.

Reply

[-]

filip-z@reddit

What speeds are you getting with SGLang? I'm using vLLM now and get about 90 tk/sec generation with empty context, goes down to maybe 65 to/sec with context at 150K tk.

Reply

[-]

Sofakingwetoddead@reddit

Running a 6000 pro workstation edition - lower context 125 tk/s generation and probably average around 5k pp with the peak around 8k at lower context. 200k context we do around 95 to 100 tk/s in generation. Cutlass fp8 backend, eagle mtp spec num 3 , topk 1, draft tokens 4. On 2 concurrent jobs we do around 190 tk/s generation. Ingestion still increases by maybe 70% or so. Frankly, the ingestion is sooo fast that I don't even pay attention anymore. We can go on a single prompt from 0 to 150k context in maybe 20 seconds when spawning numerous read workers.

Reply

[-]

_madar_@reddit

Hmm, I think I need to look at my setup again, I'm on a 6000 pro workstation as well, with vllm under normal usage I'm getting 45/s. I stress tested 8 concurrent sessions and get some good numbers there, but single session I wonder if I'm missing some performance.

Reply

[-]

luckyj@reddit

Side question, how much of thos 256k tokens are actually useful without context degradation? On my (Very anecdotal agentic) tests at Q5\_UD\_K\_XL and KV at Q8, anything over 120k tokens, it starts messing up numbers (for example, flipping two numbers from a couple of messages ago)

Reply

[-]

Sofakingwetoddead@reddit

For me, all of them. FP8 KV16

Reply

[-]

My_Unbiased_Opinion@reddit (OP)

Im not using it for coding. its for assistent work for non tech related fields. I find 262K useful because context compression (hermes agent) is far worse when I am doing long context tasks like research. for my use case, KV Q4 works decently well, albiet a bit forgetful. but I just have to remind the model once in a while. thats the main issue really.

Reply

[-]

DiscipleofDeceit666@reddit

If you try turbo kv cache compression or something, models start to fall apart within 10k context lol they’ll make sense but they’ll omit characters left and right. Like they’ll say rx680 instead of rx6800

Reply

[-]

LikeSaw@reddit

Really depends on the quant, cache etc. I only use KV bf16 and model in bf16 or Q8 (vllm or llama.cpp). For basic chatting and agentic work it still goes strong 180k+ context but coding gets really bad after around 100k context.

Reply

[-]

vevi33@reddit

Also KV Q8 causes noticeable degradation on longer contexts. Unfortunately... I had to experience fist handedly. :/

Reply

[-]

MaxKruse96@reddit

bf16 kv cache: q4 model degrades around 100k for me, q6 nodel around 160k. scales up to 256k at bf16 model quant from what i heard

Reply

[-]

fmlitscometothis@reddit

I wonder if degradation at high context is related to quant size.

Reply

[-]

MaxKruse96@reddit

yes it is

Reply

[-]

shbong@reddit

Is it really worth having a context of 262K tokens? Considering the attention span of LLMs having such big context window can be counter productive as said in the "Lost in the Middle" paper

Reply

[-]

Sofakingwetoddead@reddit

Yes. 100% worth it if you need it.

Reply

[-]

shbong@reddit

Like in what circumstances? Also if someone else has tried did you noticed a degradation in performances (the model being accurate and smart) about the things you are asking with such big context windows?

Reply

[-]

My_Unbiased_Opinion@reddit (OP)

here is one of my use cases: I have the model log into my wifes school account and scrape her classes weekly and make a calender of what she needs to finish by what deadline. and using that information, suggest what to study. she is in nursing school. so far its been great. and the 262K context is far better than the lossy context compression you would run into with a smaller context.

Reply

[-]

Sofakingwetoddead@reddit

Well, the average coding task on our codebase starts with \~120k worth of ingestion between the harness and relevant code, give or take \~30k, that's before coding and testing. Typical context swells to 200k or more quickly and we end up needing to compress context \~20% of the time. We need more context.

Reply

[-]

Automatic-Arm8153@reddit

Your terrible at AI coding workflows then. Scout the codebase first, then get that agent to give a report/prompt for next session to implement or plan further. So implement/planning ai session doesn’t have to waste context searching for which files do what and so on. No way your coding with 200K with qwen and getting anything worthwhile. Even with Claude 200k is fishy unless following a well made plan.

Reply

[-]

Sofakingwetoddead@reddit

Ya ok. So when I was building full-stack SaaS software exclusively using AI back in 2024, when everyone was saying it was impossible, I must have been pretty bad at AI coding workflows... "now way your coding with 200k" ORLY? How much would you like to wager on that? Let's get a third party to hold our cash and then I will demonstrate. But make it worth my while, please.

Reply

[-]

shbong@reddit

Interesting so mostly is harnessing, do dynamic skills fit in your stack?

Reply

[-]

Sofakingwetoddead@reddit

Only around 20k is the harness. Afterwards, there is AAR ingestion. The bulk comes from the reading the code modules. The codebase is pretty complex so the agent needs to consider downstream consequences of edits as a priority. The coder will spend a lot of time evaluating those downstream, cross-module dependencies. We do a dynamic skill thing too. It's part of the harness instructions. Like how to identify skill candidates and propose them.

Reply

[-]

AndreVallestero@reddit

Q4 KV, oh boy... To answer your question, it should be around 64GB

Reply

[-]

My_Unbiased_Opinion@reddit (OP)

Ive had good luck with KV at Q4 after the attn-rot enhancements. No issues with my hermes agent. but I know Q8 or 16bit is better for sure.

Reply

[-]

DE4DLY_UNIKORN@reddit

All of it yeeeeesssss

Reply

[-]

viperx7@reddit

I have llama.cpp running qwen 27B Q8 with image support and MTP 4 with 220k ctx (kv cache isn't quantised) With 48GB VRAM if you drop image support you can squeez 250k CTX If you drop MTP then you can get 262k ctx with image support on same hardware But in my opinion sacrificing 42k ctx for twice the speed is worth it

Reply

[-]

Prudent-Ad4509@reddit

I've seen figures that it is just shy of 41Gb for Q8 with 16bit context. But most calculators are wrong and not all Q8 quants are the same. 64gb is definitely enough for both 27B and 35B (separately), and you can usually drop about 30% of context if needed to fit 48gb. Whatever is the case, you need to forget words "ollama", "lm studio" and others like it first.

Reply

[-]

motorcycle_frenzy889@reddit

I’m running the FP8 version with FP8 KV cache and it uses just under 48GB to serve full context on vLLM

Reply

[-]

ovrlrd1377@reddit

try this: [https://www.reddit.com/r/Qwen\_AI/comments/1tnz5l9/most\_vram\_calculators\_are\_wrong\_about\_qwen\_36\_its/](https://www.reddit.com/r/Qwen_AI/comments/1tnz5l9/most_vram_calculators_are_wrong_about_qwen_36_its/)

Reply

[-]

lordekeen@reddit

In your current setup you could reduce the context to 128k with q8\_0/q5\_1 kv instead o q4, that alone would give you better results imediately.

Reply

[-]

anthonyg45157@reddit

This is what I'm doing currently with my 3090 and was able to get q5 at 115k and the image model loaded in GPU was well. New CPP builds really help maximize vram

Reply

[-]

Alternative_Ad4267@reddit

Realistically? 64GB VRAM, it will fit, but above 90% sometimes. And for the BF16 one, around 100GB VRAM.

Reply

[-]

Even-Lecture352@reddit

I'm running the official 27B FP8 model on vLLM with a modified RTX 4090 48GB — this gives me a 258K context window. Here are my parameters: docker run -d \--restart always \--name vllm-qwen \--gpus all \-v \~/.cache/huggingface:/root/.cache/huggingface \-v \~/vllm-templates:/templates \--env HF\_TOKEN="$HF\_TOKEN" \--env PYTORCH\_CUDA\_ALLOC\_CONF=expandable\_segments:True \-p 8000:8000 \--ipc=host vllm/vllm-openai:latest Qwen/Qwen3.6-27B-FP8 \--port 8000 \--max-model-len 258017 \--chat-template /templates/chat\_template.jinja \--reasoning-parser qwen3 \--language-model-only \--enable-auto-tool-choice \--tool-call-parser qwen3\_coder \--speculative-config '{"method":"qwen3\_next\_mtp","num\_speculative\_tokens":2}' \--compilation-config '{"max\_cudagraph\_capture\_size":16,"mode":"VLLM\_COMPILE"}' \--gpu-memory-utilization 0.975 \--enable-prefix-caching \--performance-mode interactivity \--attention-backend flashinfer \--kv-cache-dtype bfloat16 \--default-chat-template-kwargs '{"enable\_thinking": true, "preserve\_thinking": true}' \--max-num-seqs 1 \--override-generation-config '{"temperature":0.6,"top\_p":0.95,"top\_k":20,"min\_p":0.0,"presence\_penalty":0.0,"repetition\_penalty":1.0}'

Reply

[-]

p_235615@reddit

qwen3.6:27b-mtp-q8_0 ctx 262k in ollama: /usr/lib/ollama/llama-server 47878MiB but with KV stuff 48GB could be too small.

Reply

[-]

acquire_a_living@reddit

Can run BF16 with 262K and MTP with 3x3090, 72GB.

Reply

[-]

ea_man@reddit

Well MTP will take VRAM too, quite a bit from none to -> spec-draft-n-max = 4

Reply

[-]

Shadowmind42@reddit

I'm running 48 GB of VRAM and fit the UD-Q8_K_XL with 350k context and no set to 2.

Reply

[-]

Mbo85@reddit

Hi, maybe you can take a look at this website to find the answer : [https://www.canirun.ai/](https://www.canirun.ai/)

Reply

[-]

tecneeq@reddit

Around 90 GB on a 6000 Blackwell, the model with FP16 too.

Reply

[-]

BrewHog@reddit

Just an FYI, my Mac Studio M1 ultra with 128 GB drops to about 10-13 tokens per second on that model at that quant when I reach about 80-100k context (LM Studio MLX) . Before that, it's closer to 20 tokens per second. I also have not yet tried MTP.

Reply

[-]

suprjami@reddit

128k ctx = 8 GiB 256k ctx = 16 GiB

Reply

[-]

bialad@reddit

I don't how much you need, but I have 48GB VRAM and can fit 100k context with Q8_UD_XL and MTP enabled.

Reply

Reply to Post

73 Comments