Getting error " missing tensor 'blk.0.ffn_down_exps.weight' " when trying to run mixtral models on LM Studio.

Posted by kwebber321@reddit | LocalLLaMA | View on Reddit | 3 comments

Im new to Lm studio after using ollama for a bit but trying to use this model here

https://huggingface.co/TheBloke/dolphin-2.7-mixtral-8x7b-GGUF

It downloads but when i try to run it, i get this error when trying to run the model. Does anyone know how to fix this?

🥲 Failed to load the model

Failed to load model

error loading model: missing tensor 'blk.0.ffn_down_exps.weight'

[-]

yami_no_ko@reddit

I have the same issues with mistrals MoE using llama.cpp directly. Those model quants are likely made with an old version of the tools llama.cpp comes with (specifically llama-quantize) which got changed in llama.cpp after changes were made in the MoE implementation in the codebase.

So basically there are two options:

- Convert the hf model into gguf again and quantize it with a newer build of the llama.cpp-toolset

- use an older version of llama.cpp (Basically a dead end, since this means being stuck with an ancient llama.cpp build around mid 2024)

Basically the old mixtral 8x7b quants got lost in an unsolved regression and people recommend reconverting the model with the recent toolset.

kwebber321@reddit (OP)

thank you. ill try converting them.

If you don't want to go trough the hassle I found those to be working with recent llama.cpp builds:

https://huggingface.co/mradermacher/Mixtral-v0.1-8x7B-Instruct-GGUF/tree/main