Llama 405B running locally!

I experimented with deepseek-v2 and deepseek-v2.5 today in both exo (mlx-community 4 bit quants) and llama.cpp's rpc-server mode (Q4\_0 ggufs). I have a M3 Max Macbook with 64gb of ram and an M1 Ultra Studio with 128gb of ram (not the highest end gpu cores model though). I was only able to get 0.3 tok/s out of exo using MLX (and that was over ethernet/usb-ethernet). But on llama.cpp RPC it ran at 3.3 tok/s at least (though it takes a long time for the gguf to transfer since it doesn't look like there is a way to tell the rpc-server that the ggufs have already been loaded on all the machines in the cluster). It could be that I have something wrong with my exo or MLX setup. But I can run Llama 3 8B with MLX at 63+ tok/s for generation so I dunno what is going wrong. Kind of bums me out about being able to run a big MoE quickly in a distributed setup