LM Studio - 3 GPUs, one model per GPU as different servers

Posted by MarcusAurelius68@reddit | LocalLLaMA | View on Reddit | 15 comments

LM Studio has been really easy to use, but it seems, like they dramatically changed the interface from 0.3 to 0.4. I have 3 GPUs, and want to assign one to a Research model at port 1234, one for Writing at 1235, one for Utility at 1236. Research and Utility are CUDA and Writing is Vulkan. It looks like this was possible before but not now? Should I just move to Ollama to get this level of control? Or something else?

Reply to Post

15 Comments

[-]

m94301@reddit

I am working on exactly this! Give me a couple days and I'll send a github

[-]

MarcusAurelius68@reddit (OP)

Cool. for LM Studio or llama.cpp?

[-]

m94301@reddit

For llama.CPP. LMS is great but they are focused on single server multi model. It is the most common use case, but like you my use case is dedicated instances.

[-]

MarcusAurelius68@reddit (OP)

I love their approach to pooling VRAM to tackle large models, but my new approach is to have 3. Looking forward to seeing what you come up with!

[-]

m94301@reddit

OK, its up and you can see a pic and description here. https://www.reddit.com/r/LocalLLaMA/s/NO0y4kPtPJ Let me know if you have feedback, it sounds like we run similar frameworks so I'd love to hear what I've overlooked!

[-]

MarcusAurelius68@reddit (OP)

Cool thanks! Playing with another strategy right now but will keep in mind.

[-]

lemondrops9@reddit

You can load all into LM Studio then have the program use the model you want. Loading each one to a different gpu will be a bit of a pain as you will need to load a model then disable that gpu then enable the next one before loading the next model. Next option would be to use Llama.cpp and setup scripts for each gpu.

[-]

MarcusAurelius68@reddit (OP)

Thanks. I’m looking to run 3 models concurrently, so that will likely take 3 instances. Will play with llama.cpp next.

[-]

nickless07@reddit

You need 3 server for that, so either run LM Studio 3 times, ollama 3 times or llama.cpp 3 times and so on. It was not possible before and not now with only a single instance.

[-]

MarcusAurelius68@reddit (OP)

Thanks - I have 128GB of RAM so plenty of headroom to run 3 instances. Initially I was interested in LM Studio for the UI aspect but it’s becoming a headless server now so API calls are fine. It sounds like at this point I should just configure and run 3 instances of llama.cpp to maximize hardware.

[-]

nickless07@reddit

Oh, you can run them all on the same port, the API is capable to handle @ model requests, that is why you get the API model identifier. Load model A on GPU 1 and model B on GPU 2 and then just send the requests to the desired model on the same port is not a big deal. However if you need different ports you also need different processes.

[-]

lemondrops9@reddit

exactly!

[-]

lemondrops9@reddit

Exactly!

[-]

lemondrops9@reddit

most times you can call the model you want so its not a big deal to have multiple models loaded.

[-]

MarcusAurelius68@reddit (OP)

Looks like llmster is available on LM Studio as a headless daemon so perhaps this will work. I’m basically looking to bind each GPU to a separate process so that one agent will consult one, another a second model, etc.