Vision inference engine?

Posted by umarmnaq@reddit | LocalLLaMA | View on Reddit | 7 comments

What may be the best inference engine for vision models? Still praying for llama.cpp to support all the models, and exllama doesnt have vision capabilities as far as I know.

Reply to Post

7 Comments

[-]

Cyb4Black@reddit

Late to the Party, aber TGI von Huggingface unterstützt inzwischen MLlama als z.B. das Llama-3.2-vision

[-]

cchung261@reddit

I use ONNX for phi 3 vision. Works well and fast on a 3090.

[-]

cchung261@reddit

I use the onnx genai module from Microsoft. https://www.nuget.org/packages/Microsoft.ML.OnnxRuntimeGenAI.Managed Work mainly in .Net and local llms.

[-]

You could try mine, I try to support most models: https://github.com/matatonic/openedai-vision It's not fast like exllama2/llama.cpp, but it supports most models. You can also try vllm, which has a growing list of vision models it supports.

[-]

Lemgon-Ultimate@reddit

Yeah I'm a bit disappointed with the VLM support. I think it slowly gets better but most backends still don't support vision models. It would be awesome if I could plug any vision model in my exl2 loader and it just runs like with LLM's but it seems more complicated for different architectures. Maybe this changes when Meta drops multimodal Llama.

[-]

Vision inference engine?

Reply to Post

7 Comments

Cyb4Black@reddit

cchung261@reddit

kryptkpr@reddit

cchung261@reddit

matatonic@reddit

Lemgon-Ultimate@reddit

fish312@reddit