People with low VRAM, I have something for you that won't help.

Posted by Uncle___Marty@reddit | LocalLLaMA | View on Reddit | 32 comments

\*hug\* I'm one of your kind. I Struggle like you do but I promise you. If you get more VRAM you'll think you screwed yourself of by not getting more. VRAM is the new crack for AI enthusiasts. We're screwed because the control falls upon one major company. Whats the answer? I'm not sure but more cat pics seems like a good time passer until we gain more data. Just remember. More VRAM doesnt instantly mean better results, sometimes it just means higher class hallucinations ;) Hats off to the wonderful and amazing r/localllama community who constantly help people in need, get into WILD discussions and make the world of AI chit chat pretty god damn amazing for myself. I hope others find the same. Cheers everyone, thanks for teaching me so much and being so great along the way. Low VRAM? No problem, 2 years ago you couldnt run a damn thing that worked well, now you can download qwen3.5 and have a "genius" running on your own \*\^$!.

[-]

WhoRoger@reddit

What are you using it for? Idk about vision, but for text, try Granite. It's faster than Qwen on CPU. 1B seems comparable to Qwen 2B. Also try if it wouldn't be actually faster just on CPU without Vulkan. On my old quad Intel 8350U, all in Q6_K: Qwen 3.5 2B 6 tk/s, Granite 1B 12 tk/s; Granite 7B Also 12 tk/s, but context eval takes longer. Then there's Smollm3 3B, which is my favorite. Slower but definitely the smartest of these, and can reasonably think. 7 tk/s But with Vulkan, I get third to half less than that. That's Ollama, but I doubt the engine will make much difference. Or if you want qwen but faster, there's a "tuned reasoning" 0.8B on HF, which looks pretty good. I should probably make a post about this... Other small models than Qwen 3.5 needs love too lol

[-]

Kahvana@reddit

I'm using Qwen3.5 2B on that netbook for: \- Text extraction \- Image descriptions \- Toolcalling (openzim-mcp) \- Websearch \- Brave LEO integration \- Chatting for fun I've tried Granite Micro H (3B). It was indeed faster than Qwen3.5 2B, but it's output was lower quality overall. I've ran it on both CPU and iGPU (Vulkan, OpenCL) but Vulkan is at least twice at fast. Reason you're seeing faster speeds is because your CPU supports AVX and AVX2. Mine doesn't support either (The Pentium Silver N5000's featureset is comparable to an Intel Core 2 Quad mobile processor. And ollama makes a huge difference, it's an order of magnitude slower than llama.cpp. Granite 1B H / 350M H, Qwen3.5 0.8B, Gemma3 1B / 350M with SWA-Full are all not usable due to their size. It's simply not smart enough to run openzim-mcp on. Only model that gets close is LFM2-3B-VL and LFM2.5-1.6B-VL. I really wish LFM will make a small model that supports both reasoning and VL in one model... The model I would love the most to run is Mistral Small 3 3B Reasoning as it also supports toolcalling / reasoning / VL, but it's simply too heavy to run.

32 Comments

Even with just 8GB DDR4 2400 MHz single-channel RAM (yes, no vram!) you can run Qwen3.5 2B Q4\_K\_S (with vision encoder in FP16) and 8K context on an Intel UHD Graphics 605 at 2 T/S gen with Vulkan! And the whole system doesn't even use more than 12W during inference. Sure it's very slow, but the fact it can run on a netbook at all is to me still mind boggling.

VoiceApprehensive893@reddit

if you have 16gb you can afford 26b gemma moe with 2k context and up to 25 tokens/s if youre ddr5 and you have a good igpu/cpu

linumax@reddit

How is the AI slop and accuracy level for a 2B ? I want to get a model that can fine tune my short stories. Just concern about AI slops

Depends on your sampler settings. Adaptive-P + Top-NSigma with temp 1.0 gets you far, but you would be severely constrained by the small model side.

ItsNoahJ83@reddit

I would use a Frontier model for that

Ok thanks, that would be a good option as well

overand@reddit

What's the prompt processing speed like on that? (Granted with 8k of context, your prompt can't be TOO large, but, that's a big part of what I find frustrating even *with* a GPU, and an area where smaller models really do shine)

Suprisingly 1 t/s initially and around 0.5 t/s at 8k. With ram cache you won't notice it as much though as it doesn't have to reprocess the whole 8k.

tinny66666@reddit

Whatever it is, it's measured in seconds per token instead of tokens per second.

Academic_Pick6892@reddit

It's honestly mind-boggling to see Qwen3.5 running on a UHD 605. 2 T/S is actually usable for some background automated tasks! I’ve been doing similar stress tests with 4-bit quants on older RTX cards to see where the quality to speed tradeoff breaks. Are you using llama.cpp for the Vulkan implementation, or a different backend? Curious if you’ve tried pushing any image gen models on that setup or if the VRAM limitation is just too steep there.

viperx7@reddit

we should have a pool to see the VRAM distribution of this community .

ISoulSeekerI@reddit

Crack plus Ram=CRam

redditor_no_10_9@reddit

Imagine Intel release a GPU where anyone can add VRAM by themselves and then allow anyone to manufacture the VRAM. Probably will destroy the super green company's business modal of starving GPU of VRAM

https://preview.redd.it/r1q6jeapsdsg1.png?width=686&format=png&auto=webp&s=0e0d5cf89937e83ef581fefa22a95d1318b855de 1998 called and want it's upgradable VRAM back (pic: ATI Xpert@Work PCI)

xienze@reddit

You can have fast VRAM or socketed VRAM but not both.

Ardalok@reddit

At the moment, memory is too expensive, unfortunately.

grumpoholic@reddit

Intel is still in the foundry industry.

toooskies@reddit

I mean, they did just release a 32gb card for $1000.

This_Maintenance_834@reddit

got two already, also posted a first day impression on r/localllm

Far_Cat9782@reddit

Had to upvote because I literally used the "generate an image of a cat" for a test for my comfyui tool earlier today. That's 🤣

crsnplusplus@reddit

I do use my hx370 igpu pretending it's an entire datacenter. And I have a lot of patience (and ram, bought 96gb for three strawberries when it was possible)

Impressive_Author_74@reddit

Hello , I am a homeless hobo trying to run qwen3.5 on a 100buck 6w Intel n100 for document sorting and your reddit thread made feel recognized. I m happy to leave this world knowing that for a brief a moment in the history of this sub, among all those rich tech enthusiasts, I have existed. 😭

JackStrawWitchita@reddit

Don't even need a GPU. Plenty of AI power in CPU-only rigs: [https://www.reddit.com/r/LocalLLaMA/comments/1qxgkd1/cpuonly\_no\_gpu\_computers\_can\_run\_all\_kinds\_of\_ai/](https://www.reddit.com/r/LocalLLaMA/comments/1qxgkd1/cpuonly_no_gpu_computers_can_run_all_kinds_of_ai/)

MrPandastic@reddit

Just learned few days ago that the good folks at DeepSeek and Peking University working on something where you can run models with much lower vram and use ssd to cache stuff. Or something like that. So there is definitely hope. Paper for people smarter than me: [https://github.com/deepseek-ai/Engram/blob/main/Engram_paper.pdf](https://github.com/deepseek-ai/Engram/blob/main/Engram_paper.pdf)

kpcurley@reddit

Checking in with a Intel ARC140v with 32 GB of shared on packages LPDDR 5 with half allocated to the GPU. Feeling just fine pulling 12-15 TPS on 12b models in OV with int4 quant. I mean I have a 3080 10gb that can run 4b-7b models a good deal faster but the 140V on my Intel 288v feels a good deal more useful with the LPDDR5 (8400).

People with low VRAM, I have something for you that won't help.

Reply to Post

32 Comments

Kahvana@reddit

VoiceApprehensive893@reddit

WhoRoger@reddit

Kahvana@reddit

linumax@reddit

Kahvana@reddit

ItsNoahJ83@reddit

linumax@reddit

overand@reddit

Kahvana@reddit

tinny66666@reddit

Academic_Pick6892@reddit

viperx7@reddit

ISoulSeekerI@reddit

redditor_no_10_9@reddit

Kahvana@reddit

xienze@reddit

Ardalok@reddit

grumpoholic@reddit

toooskies@reddit

This_Maintenance_834@reddit

Far_Cat9782@reddit

crsnplusplus@reddit

Impressive_Author_74@reddit

JackStrawWitchita@reddit

MrPandastic@reddit

kpcurley@reddit

mystery_biscotti@reddit

superSmitty9999@reddit

Ok_Sprinkles_6998@reddit

Igot1forya@reddit

kulchacop@reddit