How are Intel gpus for local models

[-]

DesmondFew7232@reddit

For LLM models, Intel provides **ipex-llm**, which is always kept up to date with the upstream repository: https://github.com/intel/ipex-llm/blob/main/docs/mddocs/Quickstart/ollama\_portable\_zip\_quickstart.md. It's so easy to run using the portable version of Ollama.

Reply

[-]

LostHisDog@reddit

Use Nvidia if you want to AI. Literally every development and innovation has come to Nvidia first and sometimes Nvidia only. We aren't anywhere near the point where competing technologies are keeping pace. Unless you just want to pick a static workflow that you might eventually manage to setup on intel / amd and never change anything as stuff continues to improve daily, just buy Nvida and be done with it.

Reply

[-]

orbital_one@reddit

I wouldn't even bother with Intel GPUs unless you already know what you're doing. It can be frustrating getting things to work, even after following instructions. You'll likely have to use older kernel versions, older libraries, and specific versions of the oneAPI toolkit (did you install v2025.1.0? Oops! Only v2025.0.1 is supported...) If you can figure it out, however, it works fine given the cost of the card.

Reply

[-]

COBECT@reddit

You can find the answer on your question here https://github.com/ggml-org/llama.cpp/discussions/10879

Reply

[-]

prompt_seeker@reddit

I have a A770 and 2x B580, and I don't recommend them for LLM. They are slower than RTX3060 for LLM, and have issue about compatability. They are quite good for Image generation though.

Reply

[-]

MoffKalast@reddit

Yeah IPEX adoption is nonexistent, SYCL can seemingly only do fp16 at full speed, and Vulkan is so slow on Arc it's not worth bothering with. Maybe if Intel gets their Vulkan shit together eventually since it's the only widely used backend option out of all three.

Reply

[-]

DisgustingBlackChimp@reddit

Had nothing but issues, but I'm using UNRAID.

Reply

[-]

Calcidiol@reddit

I've certainly had my share of issues with intel & linux and AIML inference. But I started really early when ARC first came out and the HW / SW has matured a lot since then. I'd put it like this -- if someone wants "it just works" with a wide range of models and inference SW I'd suggest getting a 24 GBy VRAM NVIDIA DGPU like 3090 (maybe) or 4090 (better) or maybe whatever the upcoming 5080 variant that might be interesting is or such. It'll probably cost 2x-4x+ what the B580 does but it'll be possibly a lot faster and have possibly a lot more VRAM and possibly have much much less headaches with SW limitations and model limitations. OTOH if you put in significant effort to learn / adapt and moderate your expectations based on what is known to work well before you buy then buying one or two B580s or the upcoming B50 / B60 could be a solid intel choice. Really just depends on time saved & limitations avoided vs. money vs. performance trade-off. I'd buy a couple intel B50/B60/whatever cards if they were a fraction of the price I'd spend on a 4090/5090 for several LLM uses though for some uses if it could be cost justified I'd get the 4090/5090 or such (e.g. bleeding edge video generation models or faster image SOTA generation or whatever). But just to run qwen3 or gemma or phi4 or such models intels should work fine for most generic LLM needs. The multimodal model support is kind of lagging in llama.cpp so if the models one wants are first class supported by say vllm or some inference UI SW then I'd check for GPU / OS compatibility before investing

Reply

[-]

Calcidiol@reddit

PS several inference SW UIs / apps / engines do have optional containerized runtimes they officially support or at least have example dockerfiles / images you can learn / start from. openvino, oneapi / sycl development / runtime, llama.cpp, AFAIK ollama / webui, ramalama, docker itself may support running models on intel DGPUs (as a container engine it certainly can enable their use in containers but they've been starting to add "integrated" llm model inference configuration directly into the docker programs themselves recently but IDK if it is just for nvidia or also maybe intel / amd at this time). I'd advise taking some time to learn / experiment with llm support via containers in fedora since it has devops / administration / security (isolation / configuration independence) etc. benefits. And with a certain amount of inference / UI sw you'll see examples how to run it natively on a host but maybe only examples / ready to go configurations for ubuntu and maybe not fedora. So to some extent using a suitable container might ease any modest / significant portability issues wrt. fedora in a few cases. Also it can be annoying to have to install sometimes a bunch of python / huggingface / ipex / sycl / vulkan or whatever SW to support the inference stuff and then have that on your host OS and dealing with versions, virtual environments (venvs) for python, etc. etc. so containers make that more of a "who cares, it's only going to affect the container" thing vs. having to migrate and mix and match on the host OS config. IDK if lmstudio supports containers on linux or fedora host but that's a non foss but free inference sw. ollama is popular with the consumers. llama.cpp / ramalama / podman / docker / openvino / vllm / pytorch / diffusers / transformers etc. are popular with the more sysadmin / scripting / linux admin type users.

Reply

[-]

terminoid_@reddit

intel A770 here, i use the regular llama.cpp SYCL build, it's good. the Vulkan build had faster TG speeds up until recently, but SYCL is back on top now with fast PP and TG speeds. i mostly use Windows for LLM stuff right now, but I dual boot Ubuntu and it works fine there.

Reply

How are Intel gpus for local models

Reply to Post

10 Comments

DesmondFew7232@reddit

LostHisDog@reddit

orbital_one@reddit

COBECT@reddit

prompt_seeker@reddit

MoffKalast@reddit

DisgustingBlackChimp@reddit

Calcidiol@reddit

Calcidiol@reddit

terminoid_@reddit