Trying out gemma4:e2b on a CPU-only server
Posted by SensitiveCranberry00@reddit | LocalLLaMA | View on Reddit | 8 comments
I am running Ubuntu LTS as a virtual machine on an old server with lots of RAM but no GPU. So far, gemma4:e2b is running at eval rate = 9.07/tokens second. This is the fastest model I have run in a CPU-only, RAM-heavy system.
No_Business_1696@reddit
How much ram are we talking and why did you go for low parameter count?
dinerburgeryum@reddit
Low param count = less data to pull onto the CPU from RAM during inference. OP mentioned it was an “old” server, so we’re probably talking about DDR4; even slower.
EffectiveCeilingFan@reddit
DDR4 is considered old now 😭😭😭? I thought OP was talking like DDR3.
dinerburgeryum@reddit
I think DDR4 is like what, 10-12 years old at this point? So yea, I mean, I guess I'd consider it relatively old in hardware terms.
EffectiveCeilingFan@reddit
10 years ago?! Damn I’m gettin old 🧙♀️
dinerburgeryum@reddit
lol same buddy 👴
SensitiveCranberry00@reddit (OP)
128 GB RAM in the server, 72 GB allocated to this virtual machine. If you are running htop in a terminal window, you can see the model loading into RAM.
pmttyji@reddit
I see that you're enjoying this model. But check Ling-mini-2.0