New Release of ROCm based MLX LLM Engine - lemon-mlx-engine
Posted by GeramyL@reddit | LocalLLaMA | View on Reddit | 31 comments
Hey everyone lemon-mlx-engine just got done integrating TheRock / ROCm 7.13 into the lemon-mlx-engine which means you get to try the latest ROCm on your local hardware with the MLX engine! This also includes various bug fixes and kernel fixes we have been seeing in Qwen3, 3.5 and 3.6 MoE and dense. try it out! https://github.com/lemonade-sdk/lemon-mlx-engine/releases/tag/b1034-stable
wizard_of_menlo_park@reddit
I thought MLX was only for apple M series chips.
How does rcom fit in here?
GeramyL@reddit (OP)
Im making a driver for amd GPUs to work on Mac over tb5 also I just didn’t announce it yet. All initial hardware initialization and firmware loading now works basically just have to test the ring and once that works I’ll port rocm libraries to work on mac. https://github.com/lemonade-sdk/mac-amdgpu
wizard_of_menlo_park@reddit
This is interesting!
Would this be like a distributed inference engine with amd egpu running heavy computationallly expesnsive layers while mac provides a lot of cheap unified memory ?
GeramyL@reddit (OP)
Haha I don’t even know what your saying right now! lol that would be CRAZY! I just thought about using one R9700 for fun lol I didn’t even think about tensor splitting and I don’t know the answer to that tb5 is supposed to be able to do 80/120 but I’ve only seen 40/40 it’s supposed to have dynamic channel diversion but it supposedly only works for video like display port chaining, but maybe Mac added some extra controls? We will see once I’m ready what is possible.
ImportancePitiful795@reddit
Not only eGPU but also APUs like the Strix Halo can run now MLX engine on x86 OS not just MacOS. 😄
wizard_of_menlo_park@reddit
Oh, i don't know enough about stix halo apus.
But this is quite intresting.
SkyFeistyLlama8@reddit
The new ONNX lol
Fit_Advice8967@reddit
Fedora support plz. Modt ppl i know who own AMD desktops (esp. halo strix community) are on fedora
GeramyL@reddit (OP)
These are fedora compatible its not a .deb file its a binary.
Fit_Advice8967@reddit
In that case I would encourage you to rename the file away from ubuntu.. Anyway glad to hear :) keep up the good work
GeramyL@reddit (OP)
We could add fedora support.
JamesEvoAI@reddit
Am on Strix Halo, run Fedora, would also appreciate this.
Ubuntu suffers from outdated packages/kernels, and Snap packages suck
Canonical has also canonically just been kind of awful
GeramyL@reddit (OP)
These are fedora compatible its not a .deb file its a binary.
soyalemujica@reddit
What's the deal with MLX in difference to GGUF ? I'm newbie
putrasherni@reddit
mlx is a big faster
putrasherni@reddit
What voodoo is this ROCm and MLX ?
GeramyL@reddit (OP)
Want more voodoo? https://github.com/lemonade-sdk/mac-amdgpu
ImportancePitiful795@reddit
Have mentioned it weeks now while in Beta 😂
Some people didn't believe that existed..... Other thought is voodoo like you 🤣
GeramyL@reddit (OP)
The best kind of voodoo lots of hard work lol
genpfault@reddit
What's the tok/s decode look like vs. llama.cpp's Vulkan backend for AMD hardware on Linux?
spaceman_@reddit
Is this work being upstreamed somehow?
GeramyL@reddit (OP)
There is a rocm-support branch in MLX and I’m working on a amdgpu driver https://github.com/lemonade-sdk/mac-amdgpu
spaceman_@reddit
Awesome stuff, thanks for your efforts
Formal-Exam-8767@reddit
So AMD GPU cards work with Apple Silicon?
GeramyL@reddit (OP)
Haha not yet give me a few more weeks.
https://github.com/lemonade-sdk/mac-amdgpu
All hardware bring up works now I’m missing some additional things for testing the ring but nearly there. lemon-mlx-engine will be the first to test and work on Mac with a amdgpu over rocm :)
Rattling33@reddit
Niceeee!
ImportancePitiful795@reddit
Thank you 🥳🥳🥳🥳🥳
Gailenstorm@reddit
Interesting, I never thought you could run MLX on non-Apple silicon hardware. Hope it will soon make it into the lemonade server interface.
Thank you for your work!
Plane-Structure-2020@reddit
nice to see more ROCm support getting added.
GeramyL@reddit (OP)
It’s maturing very well
GeramyL@reddit (OP)
This will eventually be going into lemonade as well as getting MTP support too!