what's the cheapest hardware I can run Llama 3.2 11b (image inference) on?

Posted by dirtyring@reddit | LocalLLaMA | View on Reddit | 7 comments

I don't know if you'd need a whole Jetson for that, or if there are cheaper alternatives. appreciate this is a super newbie question and would be happy if you pointed me to materials intead

Reply to Post

7 Comments

[-]

AngleFun1664@reddit

$599 or $499 if you’ve got a .edu email address or are military

[-]

Zeddi2892@reddit

It it’s about image inference you cant use GGUF so you have to get at least 24 GB of vram. Still the macmini might be the best cheap solution here

[-]

isr_431@reddit

Ollama supports quantisized Llama 3.2 vision

[-]

Weary_Long3409@reddit

This might be not 11b as your requirements. But Qwen2-vl-7b is amazing. It runs >50 token/sec using vLLM on a cheap 3060.

[-]

I'm running Lllama3.2:11b with Ollama on Tesla M40 gpu, which costs $80 on ebay (+shippong +tax if applicable to you). If you have an old PC with enpty PCIe slot laying around, then it's going to be the absolutely cheapest way.

[-]

kif88@reddit

Someone ran an 8b vision model on a Rockchip SBC recently. 16b orange pi 5 pro is about $130 , it might do it. https://www.reddit.com/r/LocalLLaMA/s/2YUfFVX0SH