How to improve response times for multimodal requests?
Posted by coolahavoc@reddit | LocalLLaMA | View on Reddit | 2 comments
I am running Gemma 3 12B on my local computer. My prompt is about 1000 tokens of text + 3-4 images. My computer is just a regular AMD CPU (no GPU) + 64GB of DDR5 RAM, so understandably the response is slow. Particularly I have noticed that it takes more time to just process my input.
My question is what hardware would help improve this:
1. Obviously a GPU would help - but what should I look for in a GPU to get better response times?
2. Would the newer AMD Ryzen™ AI 9 HX 370 APU help or would I need to go for an AMD Ryzen AI Max+ 395 APU's?
3. If I got for the AMD Ryzen™ AI 9 HX 370 APU, some PCs come with upgradeable RAM i.e. DDR5 (going up to 96GB), while others come with faster LPDDR5 RAM - but with the caveat that the max RAM is capped at 64 GB. I want to be able to run slightly larger models on it (e.g. Gemma 3 27B), but not sure if I need to go for the LPDDR5x versions.
Background-Ad-5398@reddit
have you tried qwen 30b a4b, if your just running cpu, it might be the better choice. unless your using it for vision, then nvm
coolahavoc@reddit (OP)
Yes, I am using it for vision.