Llama4 is probably coming next month, multi modal, long context

Oh, the no support for vision on my comment was about the MLX side and exclusively to gemma. But mlx-vlm is working on that and adding the new mistral as well. I actually like llama.cpp and hope they do well, and get all the refactoring they need in place. I, for one, would rather have a wealth of options that all work. But you’re correct that so long as an engine supports proper OpenAI api standards, what we use barely matters. My pipe dream right now is to see a unified way to unload models via the api. Right now ollama does it the best, by passing a request with the model name and no prompt with keep_alive set to false. I haven’t tested if the model name is strictly required because I’m not using ollama, but if no model name was required (therefore unloading the current model, no matter which one) it would be primo! Something like a post to `/v1/models/unload` to unload models would make coordinating different types of models (say, loading llm, generating text, loading tts/stt and dealing with audio, loading diffusion model for image) much easier on the llm side.