[-]

Porespellar@reddit

Sorry for the dumb / noob question, but is this going to allow for Windows-native VLLM? If not, what does this enable for inference-focused users? Again, I’m new at this and just trying to understand where this fits into the AI stack.

[-]

KallistiTMP@reddit

If VLLM supports windows, it would allow you to manage VLLM inference servers with Triton. Triton is a framework for managing inference servers, which can use various backend plugins including but not limited to VLLM. It's also commonly used with TensorRT.

I have no idea why anyone would ever want to use Triton or VLLM in a windows environment, that's probably a Geneva convention violation. If you are considering trying to run a windows based inference server, my advice would be don't. You will have a very bad time. Triton is geared towards at scale enterprise use cases and serious big boy deployments, you don't need it if you're just messing around on your local PC, and Windows is completely unfit for any production serving use cases.

[-]

Porespellar@reddit

Is an A100–enabled VM “big boy” enough for me to consider using this and would it run on WSL? I ask because, my org is not allowed to run straight up Linux unless it’s WSL.

[-]

KallistiTMP@reddit

I ask because my org is not allowed to run straight up Linux unless it’s WSL.

I am so sorry. They won't let you run a Linux VM even?

I honestly don't know what to suggest short of applying for an exception or finding a new company. I have never heard of anyone crazy or dumb enough to even attempt to run a production LLM backend on Windows machines, even Microsoft doesn't use Windows for that.

There are instructions for building Triton for Windows 10 in the official NVIDIA docs. They might work, if you are lucky. Your performance will likely be garbage, as will your security and stability, but it is in theory possible, and if that's your org policy I'm sure you know that and have lowered your expectations appropriately.

Please, for your own sake, if they ask you to deploy that outside of an air gap, warn them that the server will get hacked and used to mine crypto, and get it in writing that your boss is aware of the risks and wants you to do it anyway. Windows is not fit for production serving use cases.

As someone explained below I was mistaken by the way, this post is actually about OpenAI Triton, a product that has an unfortunate naming collision with NVIDIA Triton but is otherwise completely unrelated.

[-]