Building a Mini PC for aya-expanse-8b Inference - Recommendations Needed!

Posted by Whiplashorus@reddit | LocalLLaMA | View on Reddit | 35 comments

Hello everyone, I'm an artificial intelligence enthusiast and I'm looking to build a mini PC dedicated to AI inference, particularly for machine translation of novels and light novels. I recently discovered the Aya-Expanse-8B model, which offers exceptional performance in English-to-French translation. My goal is to build a mini PC that can do very fast and energy-efficient inferencing to load models from 8B to 27B (up to the Gemma2-27B model). I'm aiming for a minimum of 40-50 tokens per second on the Aya-Expanse-8B model, so I can do light novel or novel machine translation efficiently. I'm aware that RAM bandwidth and GPU bandwidth are key factors for AI inference. So I'm looking for the best recommendations for the following components: * CPU with an IGPU or NPU that would be relevant for AI inference. I don't know much about NPUs, but I'm wondering if it might allow me to do something functional at high speed. Can you give me some information on the pros and cons of NPUs for AI inference? * RAM with high bandwidth to support large AI models. I've heard of the Smokeless-UMAF GitHub project that allows a lot of RAM to be allocated in the form of VRAM to the IGPU. Could this be a good solution for my configuration? * Other components that could have an impact on AI inference performance. I'm also looking for mini PCs with good cooling, as I plan to run my system for extended periods (4h to 8h continuously). Can you recommend any mini PCs with efficient cooling systems? I'd be delighted to receive your answers and recommendations for building a mini PC dedicated to AI inference. Thanks to the community for your advice and experience!

Reply to Post

35 Comments

[-]

Downtown-Case-1755@reddit

If you can wait a bit, AMD's upcoming strix halo APUs will blow away other Mini PCs in early 2025. > maybe I'm crazy, but do you think it would be possible to run aya-expanse-32b with more than 25token/s on a mini pc (with quantization of course)? With batched calls in something like vllm? *think* I don't think so, and it would be hard to get working, but TBH it could be much better than the serial performance most people test on these APUs.

[-]

zerostyle@reddit

Double the memory bandwidth will help with these a lot as well.

[-]

Downtown-Case-1755@reddit

It's well over double, and that's the main selling point!

[-]

schlammsuhler@reddit

Well asking for 40-50t/s is actually a lot. I have a 4070 super and get 40t/s for llama3.1 8b Q4KM. Qwen is the fastest for the size

[-]

zerostyle@reddit

Which Qwen model do you recommend for a 32gb macbook M1?

[-]

schlammsuhler@reddit

14B instruct. You would probably aim for mlx at 4-6bpw

[-]

zerostyle@reddit

If I’m using LMstudio any reason to use mlx vs gguf? I don’t really know the difference

[-]

matadorius@reddit

Mac mini m4 retails around 500$ not sure if you can find any cheaper

[-]

Whiplashorus@reddit (OP)

What performance can I expect? I'm not a fan of Apple things, but their hardware are the best for energy efficiency. I read a lot of things about the Mac mini M2 Max series, which are better than the new M4 for inference. Are there good comparisons or benchmarks to let me know where I'm going?

[-]

matadorius@reddit

If you want a mini pc they should be the best in the market

[-]

Whiplashorus@reddit (OP)

what model ? M4 or M2 max (for my usecase)

[-]

Mysterious_Finish543@reddit

I think between M4 and M2 Max, M2 Max is the way to go. There are 3 main factors to think about when picking hardware for LLM inference: 1. Compute 2. Memory size 3. Memory bandwidth. The M2 Max crushes the M4 on all 3 of these metrics. 1. Compute: Even M4 Pro loses to the M2 Max in raw compute. It has slightly above half the number of GPU cores, and the architectural improvements don't make up for 18 cores. Given that the M4 Pro loses to M2 Max, M2 Max will crush the base M4. 2. Memory Size: The M2 Max can go up to 96 GB of unified memory for models, 3x the M4. 3. Memory bandwidth: The M2 Max has 400 GB/s compared to the M4's 120 GB/s. It's not even close. Personally, I daily drive a **binned M2 Max (12+30)** with 32 GB of RAM. Here are some benchmarks: | Model | Tokens/Second | |----|----| | Qwen-2.5-7B-Instruct-Q8\_0 | 30 | | Qwen-2.5-14B-Instruct-IQ4_XS | 20.5 | | Qwen-2.5-32B-Instruct-IQ4_XS | 12 | Since my M2 Max is binned, I can see the full 38 GPU core M2 Max doing ~40 tokens/second for an 8B model. That being said, I find ~30 tokens/second more than enough for me. At that speed, it's already generating tokens faster that I can read them.

[-]

zerostyle@reddit

Curious why the Qwen model? And a 32b parameter model is fitting into 32gb for you?

[-]

NEEDMOREVRAM@reddit

I don't understand "Binned M2 Max." While I understand the concept of "binned"?

[-]

Mysterious_Finish543@reddit

Binned means cut down. Some chips are partially defective, so the faulty IP blocks are fused off and the chip is sold for cheaper. The full M2 Max has 38 GPU cores, while the binned versions has 30 GPU cores. Back then, I think this was \~$300 cheaper.

[-]

SignificantDress355@reddit

I would go for M4 Mac Mini. 16GB of Memory in the base model.

[-]

Whiplashorus@reddit (OP)

am gonna check this thanks for the suggestion

[-]

MoffKalast@reddit

Yeah this is basically the only energy efficient option right now. Apple has ARM chips with low power consumption, high memory bandwidth, and Metal which has very good inference support, and the only NPU that actually kinda works with LLMs. If you're in the US, it's probably also practically the cheapest option too. The downsides are well, mandatory MacOS. But other options come with a list of downsides longer than the average EULA.

[-]

matadorius@reddit

I don’t know if a Mac Studio is considered a mini pc you might can just build a regular one

[-]

zerostyle@reddit

May would work for 8b models, but extra ram costs a fortune on macs if he wants to do bigger models. 64gb of ram is only available in m4 pro models (for 27b models). 48gb might do the trick but you're still looking at $1839 for an m4 pro, 48gb, 1tb machine. Nothing like $500.

[-]

FullOf_Bad_Ideas@reddit

I don't think igpus and NPUs are relevant as long as they use slow RAM. I would aim for something where you can do batch inference, this would speed up translating novels a lot. You can expect around 2000 t/s on Aya Expanse w8a8 quant with 3090. I don't think there are mini PC's with 3090, I just really think you should be aware than going from single batch inference to batch inference moves you from 50 t/s to 2000 t/s, it's changing the economics of it a lot.

[-]

Whiplashorus@reddit (OP)

What is batch inference Do you have any good article talking about it ? Is there any quality loss ?

[-]

chrd5273@reddit

https://www.reddit.com/r/LocalLLaMA/comments/17sbwo5/what_does_batch_size_mean_in_inference/ Basically, simultaneous processing for multiple prompts at okce. llama.cpp supports it. You may want to take a look. AFAIK there's no quality loss, if your prompts are independent.

[-]

Whiplashorus@reddit (OP)

Am using https://github.com/bookfere/Ebook-Translator-Calibre-Plugin This plugin send multiple extract of my book It send a system prompt and in the user prompt there is the extract of the book to translate If I understand, well the batch inference system could help me improve my inference speed ? Can I achieve this on an AMD GPU(my 7800xt), or maybe on a mac mini (M2 max or even m4)? Do I need a specific os to use this type of inference?

[-]

chrd5273@reddit

Batch inference is platform agnostic feature. But I don't think these GUIs support batch inference of local model. You can spin up llama.cpp server yourself and send multiple REST request. It will process your prompt simultaneously. Note that high batch number increase VRAM usage and might give you OOM. Check this page: [https://resonance.distantmagic.com/tutorials/how-to-serve-llm-completions/](https://resonance.distantmagic.com/tutorials/how-to-serve-llm-completions/)

[-]

isr_431@reddit

This is off topic, but have you tried Mistral Nemo/Ministral for English-to-French translation?

[-]

Whiplashorus@reddit (OP)

Hello, yes, I tried it. As I said in another thread, Mistral models are pretty bad for French generation; they miss a lot of tones and context that are proper for the French language. They generate comprehensible French, but it misses something. In my opinion, Qwen2.5-7b/14b are better for French.

[-]

isr_431@reddit

That's pretty interesting to hear since Mistral is a French company. It's good to hear that you've found a better option

[-]

Whiplashorus@reddit (OP)

Yes that's crazy to be honest, but to be honest cohere models are far ahead in multilingual capability

[-]

clduab11@reddit

Not sure if you saw them on the HuggingFace leaderboard, but I was quite surprised and pleased at the results of the 3B Chocolatine model by jpacifico (it’s Phi-based).

[-]

Whiplashorus@reddit (OP)

I tried it This model was impressive for his weight but I think the best balance is aya-expanse-8b

[-]

clduab11@reddit

Nice! I'm curious too; apparently I'm reading some literature that says training in French weirdly increases accuracy on the English output. Those were just 1-2 anecdotal things I read and not sure how true that *actually* is, but 8B is the biggest I can run without major CPU/RAM offload, so I'll take a gander.

[-]

loadsamuny@reddit

if you want something small and energy efficient and fast then have a look at the HP Z2 minis. Premium price though… https://www.hp.com/gb-en/shop/product.aspx?id=8T1Q1EA&opt=ABU&sel=DTP&source=google&channel=cpc&adcampaign=OLS-UK-PS-Wks-Goo-Pmax&addisttype=xpla&kw=&adid=&infinity=ict2~net~gaw~cmp~OLS-UK-PS-Wks-Goo-Pmax~ag~~ar~~kw~~mt~&gad_source=1&gclid=EAIaIQobChMIrcXbh5nhiQMVv5pQBh36DzuiEAQYASABEgJ10_D_BwE&gclsrc=aw.ds

[-]

ICanSeeYou7867@reddit

I would get a user Nvidia quadri p6000 or Nvidia 3090ti if you can afford them. They aren't too bad used. For an 8B model, you can easily run this using Q4-Q6 on a 12gb GPU. However... the more you try, the more you learn, you are going to get an itch to try larger models. I got a p6000 used for a pretty good price. And now I want 48GB vram for larger models...you always want larger models. Also. Did you know on open router there are a few powerful models that are currently free to use. (Though typically with a small context, like 8k.

[-]

Whiplashorus@reddit (OP)

I have an rx7800xt and to be honest it's my best gpu I can run all the models I want. I want a mini pc to just dedicate it for my novel translation tasks and turn it off when it's done Soo I don't need a powerful gpu (furthermore my electricity bill has been multiplied by 3 soo no big gpu 😅)