Mac Studio M3 Ultra 512GB DeepSeek V3-0324 IQ2_XXS (2.0625 bpw) llamacpp performance

Posted by WhereIsYourMind@reddit | LocalLLaMA | View on Reddit | 32 comments

I saw a lot of results that had abysmal tok/sec prompt processing. This is from the self compiled binary of llamacpp, commit f423981a.

./llama-bench -m ~/.lmstudio/models/unsloth/DeepSeek-V3-0324-GGUF/DeepSeek-V3-0324-UD-IQ2_XXS-00001-of-00005.gguf --n-gpu-layers 62 --flash-attn 0 -ctk f16,q8_0 -p 16384,32768,65536 -n 2048 -r 1 
| model                          |       size |     params | backend    | threads | type_k |          test |                  t/s |
| ------------------------------ | ---------: | ---------: | ---------- | ------: | -----: | ------------: | -------------------: |
| deepseek2 671B IQ2_XXS - 2.0625 bpw | 203.63 GiB |   671.03 B | Metal,BLAS |      24 |    f16 |       pp16384 |         51.17 ± 0.00 |
| deepseek2 671B IQ2_XXS - 2.0625 bpw | 203.63 GiB |   671.03 B | Metal,BLAS |      24 |    f16 |       pp32768 |         39.80 ± 0.00 |
| deepseek2 671B IQ2_XXS - 2.0625 bpw | 203.63 GiB |   671.03 B | Metal,BLAS |      24 |    f16 |       pp65536 |     467667.08 ± 0.00 | (failed, OOM)
| deepseek2 671B IQ2_XXS - 2.0625 bpw | 203.63 GiB |   671.03 B | Metal,BLAS |      24 |    f16 |        tg2048 |         14.84 ± 0.00 |
| deepseek2 671B IQ2_XXS - 2.0625 bpw | 203.63 GiB |   671.03 B | Metal,BLAS |      24 |   q8_0 |       pp16384 |         50.95 ± 0.00 |
| deepseek2 671B IQ2_XXS - 2.0625 bpw | 203.63 GiB |   671.03 B | Metal,BLAS |      24 |   q8_0 |       pp32768 |         39.53 ± 0.00 |
| deepseek2 671B IQ2_XXS - 2.0625 bpw | 203.63 GiB |   671.03 B | Metal,BLAS |      24 |   q8_0 |       pp65536 |         25.27 ± 0.00 |
| deepseek2 671B IQ2_XXS - 2.0625 bpw | 203.63 GiB |   671.03 B | Metal,BLAS |      24 |   q8_0 |        tg2048 |         16.09 ± 0.00 |

build: f423981a (5022)